{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 采用5折交叉验证，分别用 log 似然损失和正确率，对 Logistic 回归模型的正则超参数调优。（各50分）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.从上一题特征工程的输出档案读取本题需要处理的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.639947</td>\n",
       "      <td>0.866045</td>\n",
       "      <td>-0.031990</td>\n",
       "      <td>0.670643</td>\n",
       "      <td>-0.181541</td>\n",
       "      <td>0.166619</td>\n",
       "      <td>0.468492</td>\n",
       "      <td>1.425995</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.844885</td>\n",
       "      <td>-1.205066</td>\n",
       "      <td>-0.528319</td>\n",
       "      <td>-0.012301</td>\n",
       "      <td>-0.181541</td>\n",
       "      <td>-0.852200</td>\n",
       "      <td>-0.365061</td>\n",
       "      <td>-0.190672</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.233880</td>\n",
       "      <td>2.016662</td>\n",
       "      <td>-0.693761</td>\n",
       "      <td>-0.012301</td>\n",
       "      <td>-0.181541</td>\n",
       "      <td>-1.332500</td>\n",
       "      <td>0.604397</td>\n",
       "      <td>-0.105584</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.844885</td>\n",
       "      <td>-1.073567</td>\n",
       "      <td>-0.528319</td>\n",
       "      <td>-0.695245</td>\n",
       "      <td>-0.540642</td>\n",
       "      <td>-0.633881</td>\n",
       "      <td>-0.920763</td>\n",
       "      <td>-1.041549</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.141852</td>\n",
       "      <td>0.504422</td>\n",
       "      <td>-2.679076</td>\n",
       "      <td>0.670643</td>\n",
       "      <td>0.316566</td>\n",
       "      <td>1.549303</td>\n",
       "      <td>5.484909</td>\n",
       "      <td>-0.020496</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies   Glucose  BloodPressure  SkinThickness   Insulin       BMI  \\\n",
       "0     0.639947  0.866045      -0.031990       0.670643 -0.181541  0.166619   \n",
       "1    -0.844885 -1.205066      -0.528319      -0.012301 -0.181541 -0.852200   \n",
       "2     1.233880  2.016662      -0.693761      -0.012301 -0.181541 -1.332500   \n",
       "3    -0.844885 -1.073567      -0.528319      -0.695245 -0.540642 -0.633881   \n",
       "4    -1.141852  0.504422      -2.679076       0.670643  0.316566  1.549303   \n",
       "\n",
       "   DiabetesPedigreeFunction       Age  Outcome  \n",
       "0                  0.468492  1.425995        1  \n",
       "1                 -0.365061 -0.190672        0  \n",
       "2                  0.604397 -0.105584        1  \n",
       "3                 -0.920763 -1.041549        0  \n",
       "4                  5.484909 -0.020496        1  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "from sklearn.model_selection import GridSearchCV  # 在所有不同的超参数组合中依照我们选定的标准给出一组最优解\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# 读取数据\n",
    "train = pd.read_csv(\"FE-pima-indians-diabetes.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.sparse import csr_matrix  # sklearn 的学习器大多支持稀疏数据输入，模型训练时间可以缩短很多\n",
    "# 查看一个学习器是否支持稀疏数据，可以看 fit 函数是否支持: X: {array-like, sparse matrix}.\n",
    "y_train = train['Outcome']\n",
    "X_train = train.drop(['Outcome'], axis=1)\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression # 从 sklearn.linear_model 中导入 LogisticRegression\n",
    "lr = LogisticRegression()  # 用 lr 代替LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logloss of each fold is  [0.48805562 0.53007069 0.45629158 0.42247552 0.48388972]\n",
      "Mean of CV Logloss is  0.47615662604739944\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score  # 导入 cross_val_score 用于比较结果调优参数\n",
    "loss = cross_val_score(lr, X_train, y_train, cv=5, scoring='neg_log_loss')  # 5 折交叉验证，用负 Log 损失来评分\n",
    "print('Logloss of each fold is ', -loss)\n",
    "print('Mean of CV Logloss is ', -loss.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. logistic 回归需要调整的超参数有：C（正则系数，一般在log域（取log后的值）影响候选参数）和正则函数（也就是常称的惩罚项（penalty：L2/L1 或 L1+L2）),  目标函数为：J = C* sum(logloss(f(xi), yi)) + penalty\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同：  \n",
    "1.设置参数搜索范围  \n",
    "2.生成学习器实例（参数设置）  \n",
    "3.生成 GridSearchCV 的实例（参数设置）  \n",
    "4.调用 GridSearchCV 的 fit 方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 负 log 损失"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score=nan,\n",
       "             estimator=LogisticRegression(C=1.0, class_weight=None, dual=False,\n",
       "                                          fit_intercept=True,\n",
       "                                          intercept_scaling=1, l1_ratio=None,\n",
       "                                          max_iter=100, multi_class='auto',\n",
       "                                          n_jobs=None, penalty='l2',\n",
       "                                          random_state=None, solver='liblinear',\n",
       "                                          tol=0.0001, verbose=0,\n",
       "                                          warm_start=False),\n",
       "             iid='deprecated', n_jobs=-1,\n",
       "             param_grid={'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
       "                         'penalty': ['l1', 'l2']},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "             scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV  #\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "penalties = ['l1','l2']\n",
    "Cs=[0.001, 0.01, 0.1, 1, 10, 100,1000]\n",
    "tuned_parameters = dict(penalty=penalties, C=Cs)\n",
    "\n",
    "lr_liblinear = LogisticRegression(solver='liblinear')\n",
    "grid_neg_log = GridSearchCV(lr_liblinear, tuned_parameters, cv=5, scoring='neg_log_loss', n_jobs=-1, return_train_score=True)\n",
    "grid_neg_log.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "交叉验证后的 log 损失评估的最佳得分：0.4759698618387177\n",
      "交叉验证后的 log 损失评估的最佳超参数为：{'C': 1, 'penalty': 'l1'}\n"
     ]
    }
   ],
   "source": [
    "print('交叉验证后的 log 损失评估的最佳得分：{}'.format(-grid_neg_log.best_score_))\n",
    "print('交叉验证后的 log 损失评估的最佳超参数为：{}'.format(grid_neg_log.best_params_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 采用 log 似然损失为模型评价指标时：最佳超参数为 C=1，采用 L1 正则。此时的负 log 损失最小，为 0.4759671838445653"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 正确率(accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score=nan,\n",
       "             estimator=LogisticRegression(C=1.0, class_weight=None, dual=False,\n",
       "                                          fit_intercept=True,\n",
       "                                          intercept_scaling=1, l1_ratio=None,\n",
       "                                          max_iter=100, multi_class='auto',\n",
       "                                          n_jobs=None, penalty='l2',\n",
       "                                          random_state=None, solver='liblinear',\n",
       "                                          tol=0.0001, verbose=0,\n",
       "                                          warm_start=False),\n",
       "             iid='deprecated', n_jobs=-1,\n",
       "             param_grid={'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
       "                         'penalty': ['l1', 'l2']},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "             scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_accuracy = GridSearchCV(lr_liblinear, tuned_parameters, cv=5, scoring='accuracy', n_jobs=-1, return_train_score=True)\n",
    "grid_accuracy.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "交叉验证后的正确率评估的最佳得分： 0.7747644512350395\n",
      "交叉验证后的正确率的最佳超参数为： {'C': 0.1, 'penalty': 'l2'}\n"
     ]
    }
   ],
   "source": [
    "print('交叉验证后的正确率评估的最佳得分：', grid_accuracy.best_score_)\n",
    "print('交叉验证后的正确率的最佳超参数为：', grid_accuracy.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 采用正确率为模型评价指标时：最佳超参数为 C=0.1，采用 L2 正则。此时正确率最高为 0.7747644512350395"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 绘图，方便更好显示比较结果\n",
    "### 3.1 # 绘制交叉验证误差曲线 - log 似然损失(负 log 损失)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L1 Test\n",
      "C=0.001 \t - Logloss=0.6931471805599453 \n",
      "C=0.01 \t - Logloss=0.6359675905837048 \n",
      "C=0.1 \t - Logloss=0.4785179561104008 \n",
      "C=1 \t - Logloss=0.4759698618387177 \n",
      "C=10 \t - Logloss=0.4764049562120358 \n",
      "C=100 \t - Logloss=0.47646699009180954 \n",
      "C=1000 \t - Logloss=0.4764758966489987 \n",
      "L2 Test\n",
      "C=0.001 \t - Logloss=0.6276647344966861 \n",
      "C=0.01 \t - Logloss=0.5149487574354622 \n",
      "C=0.1 \t - Logloss=0.4767617321863817 \n",
      "C=1 \t - Logloss=0.47615970944434044 \n",
      "C=10 \t - Logloss=0.47643826621949587 \n",
      "C=100 \t - Logloss=0.4764717527567669 \n",
      "C=1000 \t - Logloss=0.4764751617570465 \n",
      "\n",
      "L1 Train\n",
      "C=0.001 \t - Logloss=0.6931471805599453 \n",
      "C=0.01 \t - Logloss=0.6346299116732262 \n",
      "C=0.1 \t - Logloss=0.46988129497104403 \n",
      "C=1 \t - Logloss=0.46285718885587174 \n",
      "C=10 \t - Logloss=0.4627309001953993 \n",
      "C=100 \t - Logloss=0.46272952407248014 \n",
      "C=1000 \t - Logloss=0.4627295100370555 \n",
      "\n",
      "L2 Train\n",
      "C=0.001 \t - Logloss=0.6265234726992036 \n",
      "C=0.01 \t - Logloss=0.5097149885501805 \n",
      "C=0.1 \t - Logloss=0.46565315115927863 \n",
      "C=1 \t - Logloss=0.46277622377031546 \n",
      "C=10 \t - Logloss=0.4627300079648692 \n",
      "C=100 \t - Logloss=0.46272951426859976 \n",
      "C=1000 \t - Logloss=0.46272950897384535 \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'neg logloss')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xUZfb48c+ZZNIDCWmUIM0EARUERBFEUVEUpAVsuxZs667dryiuu+piY7uu688uCqKAHVHBymKjE5QivYWSEGogCSlzfn/MAAEGmJThZpLzfr3uK7c8z51zLXPm3vsUUVWMMcaYw7mcDsAYY0ztZAnCGGOMX5YgjDHG+GUJwhhjjF+WIIwxxvgV7nQANSU5OVlbtmzpdBjGGBNS5s2bl6+qKf6O1ZkE0bJlS+bOnet0GMYYE1JEZN3RjgX1EZOI9BWRZSKyUkRG+jn+bxHJ9i3LRWRnhWPXi8gK33J9MOM0xhhzpKDdQYhIGPA80AfIAeaIyGRVXbK/jKreW6H8ncAZvvVGwKNAV0CBeb66O4IVrzHGmEMF8w6iG7BSVVeragkwARh4jPJXA+/41i8BvlTV7b6k8CXQN4ixGmOMOUww30E0AzZU2M4BzvJXUERaAK2Ab45Rt5mfercCtwKcdNJJ1Y/YGBMySktLycnJobi42OlQQkJUVBTp6em43e6A6wQzQYiffUcb+Okq4D1VLa9MXVV9GXgZoGvXrjaolDH1SE5ODvHx8bRs2RIRf18ZZj9VZdu2beTk5NCqVauA6wXzEVMO0LzCdjqw6Shlr+Lg46XK1jXG1EPFxcUkJSVZcgiAiJCUlFTpu61gJog5QIaItBKRCLxJYPLhhUSkLZAI/FRh9zTgYhFJFJFE4GLfPmOMOcCSQ+Cq8s8qaAlCVcuAO/B+sS8FJqnqYhEZJSIDKhS9GpigFcYdV9XtwON4k8wcYJRvX40rKStj2KQ/MjdnVTBOb4ypRa586SeufOmn4xc0QJD7QajqZ6qaqaptVPVJ375HVHVyhTKPqeoRfSRU9XVVPdm3jAlWjLM2LOfXPV8x/Ivr+PTXecH6GGNMHRQXF3dgvW/fviQkJNC/f3+/ZW+//XY6depE+/btiY6OplOnTnTq1In33nuvUp85f/58pk6dWq24A1Xvx2I6t1V7/t3rZQThwR9v4+XZXzgdkjEmBI0YMYJx48Yd9fjzzz9PdnY2n332GW3atCE7O5vs7GyGDh1aqc+xBHGCXXRyJ9667C3cJPCfxQ/w2NdvOx2SMSbEXHjhhcTHx1ep7ooVK7jkkkvo0qULvXr1Yvny5QBMmDCBU089lY4dO9K7d2+KiooYNWoU48ePr9LdR2XVmbGYquv0xi2ZMnQCQz64ifc2jGbjR/m8PPBOewlmTAj4yyeLWbJp93HLLdnsLRPIe4j2TRvw6OUdqh1bIG699VZeffVV2rRpww8//MAdd9zBF198wV/+8hemT59OWloaO3fuJDo6mkceeYRFixbxzDPPBD0uu4OooFmDJL68ejxp4Wcwc9crDHr7z5SUlR+/ojHGVNHOnTuZOXMmWVlZdOrUidtvv51Nm7yt+nv06MF1113Hq6++isfjOeGx2R3EYRpExjL16le56oMHWF74MX3GbeOjK/9FYky006EZY44i0F/6++8cJv6uezDDqRRVJTk5mezs7COOvfLKK8yaNYspU6bQsWNHfv755xMam91B+OEOc/Pe0H/RM/kqtru+5+K3b2Ld9p3Hr2iMMZWUmJhIkyZN+PDDDwHweDwsXLgQgNWrV3P22Wfz+OOPk5iYyMaNG4mPj6egoOCExGYJ4ihEhBf6PcwVLe+mKHwRA96/lnkbcpwOyxhTS5177rkMGzaMr7/+mvT0dKZNC7xv74QJE3jxxRfp2LEjHTp0YMqUKQDce++9nHbaaZx22mlcdNFFnHrqqVxwwQUsXLiQM844I+gvqaVC/7SQ1rVrVw3WhEGvL5jMvxc+CqWNePqc5+jfoX1QPscYE7ilS5fSrl27StWpjY+YTiR//8xEZJ6qdvVX3t5BBODGMwbQOK4RI7+/j5E/3crmgr9xy9nnOB2WMaaS6mtiqCp7xBSgyzJ68kbfN3CHC88uuZtHpk2mrtx9GWOMP5YgKqFzk/Z8MOhtYsIS+WDTo9zy3huUeyxJGGPqJksQldQqoTlTr5hIckQrZu79N4PH/YOiEusrYYypeyxBVEGj6EQ+u+It2sR2YQ1jueTNh8kvsFmtjDF1iyWIKopxx/Be1sucmXQJOyI+pe/4u1mVt8vpsIwxxzKmn3cxAbEEUQ1ul5vX+v2d/iddy77oHxn8/m3MWrPF6bCMMSfI/uG+s7Oz6d69Ox06dOD0009n4sSJR5QNxeG+rZlrNYkIT/d+gCZzUnhl8b+5cdotPNn9nwzqmOl0aMaYEyQmJoaxY8eSkZHBpk2b6NKlC5dccgkJCQkHyjz//PMArF27lv79+/sdWiMQ8+fPZ9GiRfTt27dGYj8Wu4OoIXedOZxHzn4KV1QOD8/6Pf+dMcfpkIwxJ0hmZiYZGRkANG3alNTUVLZu3RpwfRvuux4Ydkp/0mKTuOubu3lh+b1s3PUoT/a7CJfLhgw3Jqg+Hwlbfjl+uS2+we4CeQ/R+DS4dHSlQ5k9ezYlJSW0adMm4Do23Hc90at5d8b3e5PoCOGTrQ9zwzuT2GdDhhtTL2zevJlrr72WMWPG4HIF9vVqw33XMx2S2/HR4He4avJNzC8ZzeA3tvPOb26mYbTb6dCMqZsC/aW//85h+Kc1HsLu3bvp168fTzzxBGeffXbA9Wy473ooPT6dyVnv0DyuDevDX+Cy1//Bxp1FTodljAmCkpISBg8ezHXXXcewYcMqVdeG+66nGkU14v1BYzmtUTd2x71N/7GPsmijzSthTF0zadIkZsyYwRtvvHGg+WplWinZcN9BFszhvqur1FPKfV//iembPkN3def5vqM4L7Ox02EZE9KqMtx3MB8xhYLKDvdtdxAngNvl5j8XjebKjOuQhj/x+y/u5Z3Zq5wOy5j6Z/in9TY5VIUliBNERPjTOSO4+4z7CYtfxKh59/C3L7JtyHBjTK1lCeIEu/n063mqx2jcMRt4Y8393PPu/ygtP/HN14wx5ngsQTjg8pP78VKfF4iK3s1Xu//Eb96czN59ZU6HZYwxh7AE4ZDuTb0d6uKjhCU8zaDXxpNnQ4YbY2oRSxAOapfUjncHvk1qTCKbo5+l/6svszLvxLRvNqY+Gj51OMOnDnc6jJBhCcJhzeOb8+7A8bRu2JqixNcYMu4ZZq/Z7nRYxpgA7B/uG6Bv374kJCTQv3//49bbtm3bgf4SjRs3plmzZge2S0pKAv784cOHs2zZsirFHoigJggR6Ssiy0RkpYiMPEqZK0RkiYgsFpG3K+wvF5Fs3zI5mHE6LSk6iQmXj6Vz6plo8iSu/+Appizc5HRYxphKGDFiBOPGjQuobFJSEtnZ2WRnZ3Pbbbdx7733HtiOiIg4UE5VjzkG05gxY2jbtm21Yz+aoCUIEQkDngcuBdoDV4tI+8PKZAAPAT1UtQNwT4XDRaraybcMCFactUWMO4ZXL3mBPiddSnjyNP7vm8d4ecYKawZrTIi48MILiY+Pr/Z5Vq5cyamnnsptt91G586d2bx5M7feeitdu3alQ4cOjBo16kDZnj17kp2dTVlZGQkJCYwcOZKOHTvSvXt38vLyqh1LMAfr6wasVNXVACIyARgILKlQ5hbgeVXdAaCq1b+iEOYOc/OP80fz9zkpvLV0LP9a+Cg5O0fwaP9OhNmQ4cYc1V9n/5Vft/963HL7ywTyHuKURqfwYLcHqx1bVSxZsoQxY8bw4osvAjB69GgaNWpEWVkZvXv3ZujQobRvf8jvbXbt2sV5553H6NGjue+++3j99dcZOdLvg5uABfMRUzNgQ4XtHN++ijKBTBH5QURmikjFKZKiRGSub/8gfx8gIrf6ysytzOQctZlLXDzYbQT/1+V+3A1+4d2cx/jdW99TXGpDhhtTX7Rp04YzzzzzwPY777xD586d6dy5M0uXLmXJkiVH1ImOjubSSy8FoEuXLqxdu7bacQTzDsLfT97Dn5eEAxnA+UA68J2InKqqO4GTVHWTiLQGvhGRX1T1kPEpVPVl4GXwjsVU0xfgpBtOvZ6UmGT++N2f+LFoFFe+di9jrr2IRrERx69sTD0T6C/9/XcOY/qOCWY4B8yaNYvf/e53AIwaNYoBAwJ7Wh4bG3tgfcWKFTz77LPMnj2bhIQEfvvb31JcfGST+IrvLsLCwigrq37fqmDeQeQAzStspwOHv3nNAT5W1VJVXQMsw5swUNVNvr+rgenAGUGMtVbq17ofL/R5nuiYXawKH82glz9k/bZCp8MyxgTorLPOOvDyOdDkcLjdu3cTHx9PgwYN2Lx5M9OmTavhKI8umAliDpAhIq1EJAK4Cji8NdJHQG8AEUnG+8hptYgkikhkhf09OPTdRb1xTtNzGHvZGBrEKDsa/otBr73Nwg02ZLgxtc25557LsGHD+Prrr0lPT6+xL/LOnTvTvn17Tj31VG655RZ69OhRI+cNRFCH+xaRy4BngDDgdVV9UkRGAXNVdbKICPBPoC9QDjypqhNE5BzgJcCDN4k9o6qvHeuzavNw3zVh/e713DjtVnL3bqV88295btA1XNguzemwjHFMVYb7PtGPmGqbyg73bfNBhJD8onxu/eL3rNixnH1bhvBo7xv4zVktnA7LGEdUaT6Ies7mg6jDkqOTGXfZG5zZ+Ewim7zHX2b8l79NXXqgr4QNI2CMqUmWIEJMrDuWl/q8QN+WlxKZOpXXlj7DvRMXUFLmYcnm3SzZvNvpEI0xdUQwm7maIHGHuflrr9EkRyfx1tK3mLp1D7lv3IYHN66wUqfDM8bUEXYHEaJc4uKBMx/gvi734W7wMwtL/smODRdRXhrjdGjGmDrCEkQIExGGnzqcp3o+hTtuLRFNJ7Jj4/nsLAx8NEhj6pN1117HumuvczqMkGEJog64vM3lPH/hfwmLzCMqfSxvzQ7e8L/GmIP2D/ednZ1N9+7d6dChA6effjoTJ048Zr1QGe7b3kHUET2b9SRSm7IvIodxiz7gD71OxWUD/BlzQsTExDB27FgyMjLYtGkTXbp04ZJLLiEhIcFv+f3DfQM89thjxMXFcf/99x9RTlVRVVwu/7/lx4wJbn8Ou4OoQ8KIgbIGFLh/4LsVdWPwQmNCQWZmJhkZGQA0bdqU1NRUqjqAaH0Z7jt0jOnn/Tv8U2fjqAFuiYaozbw4czrntb3C6XCMOSG2PPUU+5Yef7jv4l+9ZQJ5DxHZ7hQa//GPlY5l9uzZlJSU0KZNm0rX3a8+DPdtTrD2TRrQoXETwohgwY5pbNpZ5HRIxtQrmzdv5tprr2XMmDFHfSwUiPow3Hdo8HjI/3IZRTtiaF4HOiGHu8Lpnd6HL8u/ZOzMZYzs28npkIwJukB/6e+/c2gxbmyNx7B792769evHE088wdlnnw3YcN8hz7N5KVpaRFzqDqdDqTHXnXYlElbCxKVTKCk7+ny2xpiaUVJSwuDBg7nuuusYNmzYgf2hPtx3vb+DKHc1In9xPA3Si0jc8gs0Ps3pkKps/wiVqkpa9Elsiv6RL5Zsof/pTR2OzJi6bdKkScyYMYNt27bxxhtvAPDGG2/QqVP17+ArDvfdunXrujPc94lUndFcV57VgdJdHlpe14zoh74ECf3moW8uGss/5v2d1vse4eNbhx2/gjEhpiqjuQbzEVMosNFcq6DV8JMIi4bcz1ejSz52OpwaMeDky3ERzq97v2JFboHT4RhTK7QYN7beJoeqsAQBhN32OSkj/kzR1kj2vPwwlB75AijUJEYlcn76BbgbLuDNn1Y4HY4xJgRZgvBJuOIKIpo3Ju/HEvS7Z50Op0Zc0/4KJKyIj1ZMZe++6rdoMKa2qSuPyE+EqvyzsgThI+HhpD78KCUF4ewY8wLs3ux0SNV2ZuMzSY1qRnnsTD7O3uR0OMbUqKioKLZt22ZJIgCqyrZt24iKiqpUvXrfiqmiuPPOI6ZLR/J/XkDDKX8m7JpXnQ6pWlzi4up2Q3l2wbO8Pns2V3drjtSBF/DGAKSnp5OTk1PlIS3qm6ioKNLT0ytVxxJEBSJC2sOPsCYri/xJ00jrNQ/SuzgdVrUMyhjEfxY8x4bSb5m//kK6tEh0OiRjaoTb7aZVq1ZOh1Gn2SOmw0S1b0/Dy/uxY3ksJRP/D0L89jU5Oplezc7DnTCPsT+tcjocY0wIsQThR8p990OYm63T1sAv7zodTrVdccpQJGwv09Z8zfa9NpmQMSYwliD8cDduTKMbb2T3+hiKxj8KJXudDqlaejTtQXJUGtJgFpPmbnA6HGNMiLAEcRRJt9xKWEIDcn/Yh373b6fDqZYwVxjD2g4hPHYlY+fMx+MJ7cdmxpgTwxLEUYTFxZJyz30U5UdSMPFF2Lne6ZCqZfDJgxGBfPme/9lkQsaYAFiCOIaEoVlEtGpB3oIY9PM/OR1OtTSJa8I5TXsQmTCPcT+tdjocY0wIsARxDBIeTtrIhygtCGPHJ1/Buh+dDqlahmUOhfBdzNj4HRu2FzodjjGmlrMEcRyxvXoRc1Y38hc3pPzDEeApdzqkKuvVvBeJkY1wN5zDO7ND+5GZMSb4LEEch4iQNnIk5SVC/vT1kD3e6ZCqzO1yMyRjMOHxvzJh/i/sKwvdZGeMCT5LEAGIateOhgMHsmNFHCUfjYLi3U6HVGVZGVmAssf9E1MXbXE6HGNMLRbUBCEifUVkmYisFJGRRylzhYgsEZHFIvJ2hf3Xi8gK33J9MOMMRMo9d0N4BFtnlsGMvzsdTpU1b9Ccbo27EZ00j3Ez1zgdjjGmFgtaghCRMOB54FKgPXC1iLQ/rEwG8BDQQ1U7APf49jcCHgXOAroBj4qIo4MIHew8F03RlFdgW+gOW5GVkYUnbDsLts7l1y2hezdkjAmuSiUIEXGJSIMAi3cDVqrqalUtASYAAw8rcwvwvKruAFDVPN/+S4AvVXW779iXQN/KxBoMSTfdTFijRHIXxKPTHnY6nCq7sMWFNIhoSGTiHN6auc7pcIwxtdRxE4SIvC0iDUQkFlgCLBOREQGcuxlQcVyHHN++ijKBTBH5QURmikjfStRFRG4VkbkiMvdEDPkbFhdLyt33ULQ1nIKvv4VV3wT9M4MhMiySAW0uJzx+MR9mL2OPTSZkjPEjkDuI9qq6GxgEfAacBFwbQD1/Ew8cPsZDOJABnA9cDbwqIgkB1kVVX1bVrqraNSUlJYCQqi8hawgRbVqTt6gR+ulIKA/NL9ehmUNRyimJmcOHCzY6HY4xphYKJEG4RcSNN0F8rKql+Pmy9iMHaF5hOx04fFqznP3nVNU1wDK8CSOQuo6Q8HDSHniA0l2w46ccmPu60yFVSZuENnRK6URc8jzG/bTWZuUyxhwhkATxErAWiAVmiEgLIJA3m3OADBFpJSIRwFXA5MPKfAT0BhCRZLyPnFYD04CLRSTR93L6Yt++WiG2Vy9iu3cnf2ki5VOfhMLtTodUJVmZWZSF5bJq98/MXbfD6XCMMbXMcROEqv5HVZup6mXqtQ7fl/px6pUBd+D9Yl8KTFLVxSIySkQG+IpNA7aJyBLgW2CEqm5T1e3A43iTzBxglG9frSAipD4wgvJ9Sv4ChelPOx1SlVzc4mJi3XHEJM9l3E/2stoYc6hAXlLf7XtJLSLymojMBy4I5OSq+pmqZqpqG1V90rfvEVWd7FtXVb1PVdur6mmqOqFC3ddV9WTfMqaK1xc0Ue3a0XDQIG/nuW/HQN5Sp0OqtBh3DP1b98MV9wufL1lF/p59TodkjKlFAnnEdKPvJfXFQAowHBgd1KhCxIHOc78kwtSHQnJ60qyMLDyUQtx8Js6xyYSMMQcFkiD2tyi6DBijqgvx38qo3nGnpZF0043sXhtO0ezvYflUp0OqtHZJ7Wif1J6E1PmMn7WOcptMyBjjE0iCmCciX+BNENNEJB7wBDes0NHoxpsIS0oid1EaOvUhKAu9xzRZGVkUuzaypXgF05flHb+CMaZeCCRB3ASMBM5U1UIgAu9jJsP+znN3UbTFQ8HPm2HWS06HVGmXtbqMqLAoGqTMY5z1rDbG+ATSismDtx/Cn0TkH8A5qvpz0CMLIQlDhhCZcTJ5Sxqj3/wN9oTWr/C4iDj6tuqLxGfzv5U5rN9mkwkZYwJrxTQauBvvMBtLgLtEJDTbdQaJhIeTOmIEpTtK2LEU+OZxp0OqtKyMLMq0GHeDhYyfbXcRxpjAHjFdBvTxNTt9He+gef2CG1boiT33XGLPOYf8X5Mon/kWbF7odEiV0jGlIycnnExS4wVMmrOB4lKbTMiY+i7Q0VwTKqw3DEYgoe5A57miUvKXJ8PnI0Oq2auIkJWRxR7WsKt8HZ8v2ux0SMYYhwWSIJ4GFojIGyLyJjAPeCq4YYWmqFNOoeHgwez4NZKSJbNg8YdOh1Qpl7e5nAhXBElNFljPamNMQC+p3wHOBj7wLd0r9ng2h0q5+y5wR5C3rDl8+QiUFjkdUsAaRjbkohYX4Ymdz/wNeSzetMvpkIwxDjpqghCRzvsXoAneEVY3AE19+4wf7rQ0koYPp2BFCYWrtsCP/3U6pEoZmjmUEs9eohMX89bM9U6HY4xxUPgxjv3zGMeUAMdjqo+SbrqRHe9OIm9ZLC2++xdyxm+gQVOnwwpI17SutGjQgt2uBXy0oCsPXXYKDaLcTodljHHAUe8gVLX3MRZLDsfgio0l5a67KNqwl4J1YfDVY06HFDARYUjGEHZ4lrNPNvPhfJtMyJj6KpB+EEP8LBeKSOqJCDBUeTvPZZC3tAm6YCJsmON0SAEb0GYA4RJO0+Y/M27mOptMyJh6KtChNl4FfuNbXgHuA34QkUCmHq2XJCyM1AdGUJq/hx0bGsPUB8ETGkNYJUcn0/uk3pRGz2bl1p3MWlNrpuIwxpxAgSQID9BOVbNUNQtoD+wDzgIeDGZwoS7u3HOJ7dGDrb/EUL56Pvw80emQAjYkYwiF5btp0GiZjc9kTD0VSIJoqaq5FbbzgEzfDG+lwQmr7kh9YASewn3krz/Z+y5i3x6nQwpI9ybdaRLbhNRm2UxbtIW8gmKnQzLGnGCBJIjvRGSKiFwvItfjnVd6hojEAjuDG17oi2rbloZDBrN9YTElW7bC9/9yOqSAhLnCGJwxmNzSXygPy2fibJtMyJj6JpAEcTswBugEnAG8CdyuqntV9bhzUxtIuetuxB1B3toO3n4RO9Y6HVJABp88GJe4aNN6MW/PXk9ZeWi8QzHG1IxAelIr8D3wDfAVMEOtWUuluNNSSbrxRgoW5VO4LQK++LPTIQWkcWxjejbrSVHkTDbv2ss3v4bWMObGmOoJpJnrFcBsYChwBTBLRIYGO7C6JunG4YSlJJO3vA26ZDKs+c7pkAKSlZHF7tLtJKeutpfVxtQzgTxiehjvbHLXq+p1QDcgNH4C1yIHOs+tzqNgWzpMfQg8tX9I7V7pvUiJTiG1WTbfrchnbf5ep0MyxpwggSQIl6pWfLawLcB65jAHOs/9koBn0y8wf6zTIR1XuCucQScPYmPxAsIjdjF+lt1FGFNfBPJFP1VEponIDSJyA/Ap8Flww6qbvJ3nHqA0dzs78jt4Z54rqv0NwQZnDMaDh1MyljFpbo5NJmRMPRHIS+oRwMvA6UBH4GVVtQ5yVRR3bk9ie/Qgf24J5Tt2wIy/Ox3ScTWPb87ZTc5mj/sHdhXtY8rPNpmQMfVBQI+KVPV9Vb1PVe9V1dCaBacWSn3gATx7i8jP7wazXoT8FU6HdFxZmVls25dLetMce1ltTD1xrPkgCkRkt5+lQER2n8gg65qotpneznM/baKkKBamPex0SMd1QfMLSIxMJLVpNgs37OSXHJtMyJi67ljDfceragM/S7yqNjiRQdZFKXfehbjd5G04HVZMgxVfOR3SMUWERXB5m8tZXTib6KhC3rK7CGPqPGuN5BB3WipJN91EwdxVFO5rCdMegvLaPbRVVkYW5VrGqW2X8/HCjewqrN3xGmOqxxKEg5JuHE54Sgp5S5qgW5fDnFedDumYWie0pnNqZ3aGf09xaTnvz89xOiRjTBAFNUGISF8RWSYiK0VkpJ/jN4jIVhHJ9i03VzhWXmH/5GDG6RRXTAwpd99F0bJ1FJR2helPw95tTod1TFmZWWwpzKFtyzzemmWTCRlTlwUtQYhIGPA8cCneOSSuFpH2fopOVNVOvqXiT+iiCvsHBCtOpzUcPJjIzEzyfirHU7gHvn3S6ZCOqU+LPsS740lqvIDVW/fy06randCMMVUXyFhM/lozbRCRD0Wk9TGqdgNWqupqVS0BJgADayrwuuJA57nNuewo7g3zxkDuYqfDOqro8Gj6te7HsoIfaRhXak1ejanDArmD+BcwAmgGpAP34512dALw+jHqNQMqTiKQ49t3uCwR+VlE3hOR5hX2R4nIXBGZKSKD/H2AiNzqKzN369atAVxK7RTXswexPXuS/+0GyqUhTB0JtfjRzdDMoZR4SjjjlFV8sSSX3N02mZAxdVEgCaKvqr6kqgWqultVXwYuU9WJQOIx6omffYd/632Cd8a60/EOJf5mhWMnqWpX4BrgGRFpc8TJVF9W1a6q2jUlJSWAS6m9Uh8YgWfvXvK39YA1M+DXT50O6ajaNmrLqUmnku/6jnKPh3dmr3c6JGNMEAQ0J7WIXCEiLt9yRYVjx/qZmwNUvCNIBzZVLKCq21R1n2/zFaBLhWObfH9XA9PxTlZUZ0VlZpKQNYTtX/9MiTsTvngYyvYdv6JDsjKzWFewiq6ZBbwzez2lNpmQMXVOIAniN8C1eOeizvWt/1ZEooE7jlFvDpAhIq1EJAK4Cu90pQeISJMKmwOApb79iSIS6VtPBnoASwK6ohCWfOedSEQEeaszvLPOzfx/Tod0VJe2upTo8Ggaps0nd/c+vl6ae/xKxpiQEshgfatV9XJVTVbVFN/6SlUtUtXvj1GvDG8CmYb3i3+Sqi4WkVEisr9V0l0islhEFgJ3ATf49rcD5vr2fwuMVtU6n35dwIAAABytSURBVCDcqakk3XQjBT8soDC6F8z4BxTUzi/eWHcsl7W6jJ93TKdpotjLamPqoEBaMWWKyNcissi3fbqI/CmQk6vqZ6qaqaptVPVJ375HVHWyb/0hVe2gqh1Vtbeq/urb/6Oqnubbf5qqvlb1SwwtScO9nedyZ4Whpfvg61FOh3RUQzKGUFxezBnt1vDDym2s2rrH6ZCMMTUokEdMrwAPAaUAqvoz3sdFJghcMTGk3HM3xUuWURDZH7LHw8b5Tofl12nJp5GRmMEW/R/uMGH8THtZbUxdEkiCiFHV2YftKwtGMMar4aBBRLZtS96XG/BEJXunJ62FzV5FhKyMLJbvXEqP9vt4b94GikpsMiFj6opAEkS+r4mpAojIUMBmjAkib+e5EZRu3MSO0ktgw0xY9L7TYfnVv3V/IsMiiU+Zx+7iMj5ZuOn4lYwxISGQBHE78BJwiohsBO4Bfh/UqAxxPXoQe+655H8yl7KE0+DLR6Ck0OmwjtAwsiF9WvRhbv7XnJzmtpfVxtQhgbZiughIAU5R1Z6qujbokRlSR9yPZ88etuV2ht0b4cf/OB2SX1kZWewp3UPndhv4ZeMuFm6o/fNsG2OOL5BWTJEicg1wN3CviDwiIo8EPzTj7TyXxfYp0ylpfBl8/wzsqn1DbHdJ60LLBi3ZWD6dmIgwu4swpo4I5BHTx3gH2SsD9lZYzAmQfOcd3pnnFsYBCl8+6nRIR9j/svrn/GwuOl35ZOEmdhaWOB2WMaaaAkkQ6ap6par+TVX/uX8JemQGqNB57tvvKWx8NSx6D9bP9F94TD/v4oABJw8g3BVOTPI89pV5eG9e7bvTMcZUTiAJ4kcROS3okZijSho+nPDUVHKnbUTjmsLnD4Kndo191CiqERc0v4Aftkyjc4s43pq5Do+n9jXNNcYELpAE0ROY55sZ7mcR+UVEfg52YOYg78xzd1P8yyIK4obB5mxY+LbTYR0hKzOLnft20umUHNZuK+SHVflOh2SMqYZAEsSlQAZwMXA50N/315xADQcN9Haee/dHPI3PhK/+AsW7nQ7rEGc3OZtmcc1Yu+8bGsVGMO4ne1ltTCgLpJnrOn/LiQjOHCRhYaQ9+AClGzeyo6gX7M2D72rXqyCXuBh88mDm5M7msjPcfLU0l827ipwOyxhTRUGbk9rUvNhzziG217nkvz2FspOHeYcD377a6bAOMejkQbjERUTiHBR4Z5aNz2RMqLIEEWLSRvhmnlvZGMIi4Is/Ox3SIdJi0+jVrBff5HzKeW0b8c6cDTaZkDEhyhJEiInMyCBh6FB2vPcxJZk3wa9TYPV0p8M6RFZmFtuKt9ExcxNbC/bxxeLaOaeFMebYLEGEoJQ77/DOPDd9GyS08I72Wl57Btjt2awnqdGpLC/8mvTEaMbNXOt0SMaYKrAEEYLCU1JIuvkmCr76hsLmN0HeEpj/htNhHRDuCmdQxiB+3PQDA7rEMHP1dlbmFTgdljGmkixBhKgDnecm/oC26AnfPFmr7iKGZAxBVQlrMIeIMBdv2WRCxoQcSxAhyhUdTco991D88y/sjhgAxTthV+35Em4W14zuTbszdf1kLj01lffn5bB3X+1JYMaY47MEEcIaDhxA5CmnsPW1SXhOvxYKNkFp7ZkzIisjiy17t9AxM5eCfWVMtsmEjAkpliBC2IHOc5s2sWPLySBh3n4RtWScpt7Ne9MoqhE/7/6CUxrHM+6ndWgtnDrVGOOfJYgQF9u9O7Hn9SL/9bfI/zXB+6jp9Utg0wKnQ8Md5mZgm4H8b8P/GNS1AUs272aBTSZkTMiwBFEHpN1/P569e9mxIob8FSmwYw283Bsm3wV7nR0wb3DGYMq0DE/sbOIiw3nLxmcyJmRYgqgDIjMySBg2jLI95RRsjIU750H32yF7PDzXGWa95FgLp1YNW9ElrQtT1nzE4DOaMOWXzWzfa5MJharFT/Vk8VM9nQ6j2urKdUBwr8USRB2RcsftIFC6qwyiGsIlT8Lvf4SmZ8DnD8BL58Ka7xyJLSsjiw0FG+iUuY2SMg/vzt3gSBxOqUtfRqZ+sQRRR4SnpOBuEE55kYdNIx+ifPduSGkL134EV74FJXvgzf7w7g2w88R+Qfdp0Yf4iHhm5X9Ot1aNGD9rvU0mZEwIsARRh4THhxHeIIxdn3zC6gED2fP9DyAC7S6H22fD+X+EZZ/Df8+E//0dSotPSFxR4VFc3vpyvlr3FUO6JrJ+eyEzVmw9Zp0rX/qJK1/66YTEZ4zxzxJEHSIiRDR003LCO7hiY9lw881sfuwxPHv3gjsazn8Q7pgDmRfDt0/A893g10/hBDQ9zcrMotRTSnHkbJLjInhr5rFfVj+ybQSPbBsR9LiMMUdnCaIOij7tNFp98D6NbryRnRMnsXrgIArnzPEeTDgJrhgL100GdwxMuAbeGgJblwc1pszETE5PPp2PVn7AFV3T+ebXPHJ21J5OfcaYI1mCqKNckZGkPTCCFm+NAxHWXXc9uU+PxlPse6zU+jy47Tvo+1fImQcvdIdpDwd1GtOszCxW7VpFp4xdALwzu/YMDWKMOVJQE4SI9BWRZSKyUkRG+jl+g4hsFZFs33JzhWPXi8gK33J9MOOsK1pc05QW1zQ9ZF9Mly60/uhDEq++iu1vvsmawUMoWrjQezDMDWff5m0W2+ka+Ol5eK4LZL8dlN7YfVv2JSY8hhmbp3DBKWlMnLOBkrLa0evbGHOkoCUIEQkDngcuBdoDV4tIez9FJ6pqJ9/yqq9uI+BR4CygG/CoiCQGK9a6zhUbS+NHHuGk11/DU1zM2quvIe/fz+Ap8fVHiEuBAc/BLV97H0F99Ht4/WLYOL9G44hxx3BZ68uYtnYaWV0bkb+nhKmLt9ToZxhjak4w7yC6AStVdbWqlgATgIEB1r0E+FJVt6vqDuBLoG+Q4qw3Ys85h9aTP6bhoEFse+kl1g67guJffz1YoFkXuOlLGPQC7FgHr1wAk++EPcducVQZQzOGUlxezC7XbE5qFGM9q42pxYKZIJoBFRvc5/j2HS5LRH4WkfdEpHll6orIrSIyV0Tmbt1ac19idVlYfDxNn3qS9Bf+H2Xbt7Fm2BXkv/ACWubrae1yeR833TnX1xv7be9jp5kv1khv7PZJ7Wmb2JYPVn7Ab846idlrt7Nsi00mZExtFMwEIX72Hd6e8hOgpaqeDnwFvFmJuqjqy6raVVW7pqSkVCvY+ia+d29aT55Mgz592Prsf1h79TXsW7XqYIGKvbGbdYapD8KLPWHNjGp9roiQlZnF0u1LOb3NHiLCXcdt8mqMcUYwE0QO0LzCdjpwyIQAqrpNVff5Nl8BugRa11RfeGIizf71T5o9829KN2xgzeAhbHt9DFpefrBQSlu49kO4cjyU7oU3L4dJ11erN3a/1v2ICoviyw0f0//0JnwwP4c9NpmQMbVOMBPEHCBDRFqJSARwFTC5YgERaVJhcwCw1Lc+DbhYRBJ9L6cv9u0zxzL8U+9SSQ369qX1lE+IPfdc8v72N9Zddz0l6ys0QRWBdv29vbF7PwzLp3l7Y0//K5QWVf7zIhpwccuL+WzNZww7M5W9JeV8tGBjpc9jjAmuoCUIVS0D7sD7xb4UmKSqi0VklIgM8BW7S0QWi8hC4C7gBl/d7cDjeJPMHGCUb58JkvDkZNL/+xxN/zqafcuXs3rgILa//TZasbmrOxrOe8DXG/sSmP6Utzf20k8q3Rs7KyOLvaV72VI2iw5NG/DWTJtMyJjaJqj9IFT1M1XNVNU2qvqkb98jqjrZt/6QqnZQ1Y6q2ltVf61Q93VVPdm3jAlmnMZLRGg4cCCtP5lMTOfO5I56nA0330zppsOe7iU0hyvehOs/gYg4mPhbGDcYti4L+LPOSD2D1g1b8/6K9/nt2S34dUsB89btqOErMsZUh/WkNkdwN25M81dfofFjj1GYvZDVAway84MPj/yF36oX/O47uPRvsGk+vHBOwL2xRYQhGUNYuHUhp7UqIj4ynHH2stqYWsUShPFLREi86kpaf/wRUaecwuY//pGc3/+B0ry8QwuGhcNZv4M75x/aG3vB+OP2xh7QZgBul5tP13xEVpd0Pv9lC/l79h2zjjHmxLEEYY4ponlzThr7JmkPjWTvTz+x5vIB7P7ssyMLxib7emN/A4kt4OM/wGt9YOO8o547MSqRC0+6kE9Wf8IVZ6ZRUu5hUj2bTMiY2swShDkucblodP31tPrwQ9wtW7Dxvv8j5557Kdvh551Bs85w4xcw6EXYuR5euRA+vuOovbGzMrPYtW8Xa4pm0b11EuNnrqfcJhMyplawBGECFtm6FS3Hjyflvvso+PprVve/nIKvvz6yoMsFna72DgJ4zh2w8B1fb+wXoLz0kKLdGncjPS79wMvqjTuL+N/yvCPPaYw54SxBmEqR8HCSb72FVu+9S3hqKjm338GmB0d6pzg9XFQDuPgJ+P1PkN4Fpo6EF8+F1dMPFHGJi6zMLOZsmcMpzYtJiY9knI3PZEytYAnCVElU27a0mjiB5D/8nl1TprD68gHeKU79ScmE334AV70NpYUwdiBMvNb7CAoY2GYgYRLGx6s/5OozmzN9+Va2eBqewKsxxvhjCcJUmUREkHLXXbScMAFXXNyhU5weUVjglH6+3th/ghVf+npjjybFHUev9F58vPJjhp3ZFJcIn5ecceIvyBhzCEsQptqiTzv1iClO986e7b+wOwrOG+Htjd32Upj+NPy3G0Oj0tlevJ1lu2dxUbtUvijtSKmGndgLMcYcwhKEqRGHTHHqcrH++hvIffrpg1OcHi6hOQx7A66fApFx9PjiSdII471Fb/Lbs1uwW2P4vuyUE3oNxphDhTsdgKlb9k9xmvePf7L9zbHsmfEdTUc/TXTHjv4rtDoXfvcdYXNfY/Ccf/CSLuDP+gQZrg68UNSHqc99R7Q7jCh3GNHuMKIjDvvrWz/8eNRhZaIiXAfWw8Psd5ExgbAEYWqcKyaGxo/8mfg+F7Hpjw+z9uprSLr5ZpLvuB1XRMSRFXy9sQe36sFLn1/NRys/ZFLk27zLmTQqP4WSEtjnEUrKlX0eYV+Zss8DBeWw3SN4EDy48KhQjgvFhQfvugdBD6x7F5fLRXh4GOFh4US4wwkPD8ftDiciPBx3hfUIt5tIt3c70u0mIjyMyEg3EeFuoiLCiYxwE+V2E+EOJyoygujIcKIiIohyhxPpDkdc4SCugwMZlpf61tX/X/X4OYbfMqoe1KMoisfjwaPqHQrFo3jUg8dTjqJQrnjUuw/f34rl1OPd9i6+83gU2L/uOXBMVdlWFgXAyrlfekPz9x/AYTsPH6JFjyhw+AmO7IF/5DiOeszjxxv4Mbc0xnuGH6Ycs1xNOOJ6a9iW0hjCBDoE4dxSV0bQ7Nq1q86dO9fpMMxhygsKyB09ml3vf0BkZiZN/zqaqHbtjlr+tq9uY2X+Up5dvZQOJTbshjGBWOo5iXajfqlSXRGZp6pd/R2zOwgTVGHx8TR98kni+/Rh85//zJphV5By+x9IuuUWJPzI//yGZgzl3o33cn9qExLL4e0bZ3p/NXvKvX8rLofs27+u/vd7yg/+Atfyo5zD/3nV46G0vJzSklL2lZVRWlZGWWkZJeXev6VlZZSVl3vXy73r5b595eXl7Nm4FEWISstARAABEe/6gW0X4tu/f6m4LeI6UNZbz+WthuvgeeTgMeFgPW+1g+X2n+tgWbzlXHIgPjlQ3vvXe8xF3ozXEITU824++C/Nz/yPcvjOI8ocusP7uceo7+cch5eR43xGxV1bvnoOgCZ97j6yTIjZ8uUzhIUrR//ZVXWWIMwJEX/++URPnkzuE0+y9dn/UPDNtzQd/TSRbdocUu685ueRFJXEzvJtxHvCvHNQOEyACN8SW4X6i5/qCUCH2/9Vg1E5Y/HMZwDo0HuIw5FUj/xvNAAdzrnM4UhqwPSngnZqe1tnTpjwxESa/fMfNHvmGUpzcvxOcep2uRl48kAKRRnySYmD0RpjLEGYE65B30to/clkYnv5pji99jpK1h0cXmNIxhA8LmH2yWKzzBnjIHvEZBwRnpxM+nPPsXvyZLY88SSrBw0mdcT9JF51FS0atKDNFuXLTi66je9GWmwaqTGppMWkkRbjW49No3FMY1JjUmkU1YgwV+3tVLfuG+9suR3+6HAgNaCuXEtduQ4I7rVYgjCO2T/FacxZZ7H5T38md9Tj7PnqK5o88QRDf/Sw+CQh6boryS3MJa8wjwV5C8gtzKXMU3bIecIlnJSYlINJJDbtQDLZv54SnYI7zO3QlRoTmixBGMe5Gzem+Ssvs/Pdd8kb/VdWDxjISW6l0R7lsjPvP6SsRz1sL95OXmEeuXtzDySP3MJccvfmsnzHcr7b+B1FZUVHfE5SVNIhdyONYxsfcWcS4445UZdtTK1nCcLUCiJC4hVXEHvOOWx+6I+kzJlDwwJYd/0NuKKjccXEIDHRuKJjcMXEkBodTeMY3/7oNriiT8PVMAZXk2gkKorCCCVf97BVd5Nbtp3c/UmkMJeNezayIG8Bu/btOiKO+Ij4I+4+Kj7WSotJo0FEgyOaZRpTF1mCMLVKRHo6J735Bt+f04GYItCyMkrzctHCIjyFhXiKvH+p0PLpWBoBjVwuOkRHexNMTIw3yUSfhEZHUhrhotgNhW4Pe8JK2e0qYZcUs01Wkk82C9hDkVvZFyHsc0OxG4iKpEFCKokJjUmOb3wgmaTGpNI4pjFpsWk0imqES6wNiAltliBMrSMuFwVxQkEcdBn/1hHHVRUtLUUrJAxPYRGeokI8hYVoUZF3e//xokJv2cKig+WLCmFvEeFbC4kpKiKqqIgEX93jKwTWAmspC/MmjeII7981blgaASVuF0RH4oqJYV+0IsCHdw6ocJHeOxC/9yEH7k4OllHx03Vsf7kDfwTlyE5n/j9TKuw+9PMO+SA5tMaOKG+rsk9H/sb/Z4SIunId4L0Wjys4d7SWIEzIEREkIgIiIghLSKjRc6vHgxYXV7hbKcJTuNebdI6SjMoLCykq2EHxnp3s27Obsr17KC8shPwiXMUFuIrA5QHWrzx4DZVovSt+mvrur1/xiL+viEA/J9ByLfaXWzU/sAq1VIv9KyF+HeC9lg0pwWkObgnC1EotS51ptiouFxLjfc9RUz67qD0Al321pMbO6ZS6ci115Trg4LUEgz0kNcYY45clCGOMMX7ZIyZTK+klDZ0OwZh6z+4gjDHG+GUJwhhjjF9BfcQkIn2BZ4Ew4FVVHX2UckOBd4EzVXWuiLQElgLLfEVmquptwYzVmGBpcUEjp0OoMXXlWurKdUBwryVoCUJEwoDngT5ADjBHRCar6pLDysUDdwGzDjvFKlXtFKz4jDHGHFswHzF1A1aq6mpVLQEmAAP9lHsc+BtQHMRYjDHGVFIwE0QzYEOF7RzfvgNE5AyguapO8VO/lYgsEJH/ici5/j5ARG4VkbkiMnfr1q01FrgxxpjgJgh/Pf8P9AcXERfwb+D//JTbDJykqmcA9wFvi0iDI06m+rKqdlXVrikpKTUUtjHGGAjuS+ocoHmF7XRgU4XteOBUYLpvsLDGwGQRGaCqc4F9AKo6T0RWAZnA3CDGa2qRDk2sH4QxTgvmHcQcIENEWolIBHAVMHn/QVXdparJqtpSVVsCM4EBvlZMKb6X3IhIayADWB3EWI0xxhwmaHcQqlomIncA0/A2c31dVReLyChgrqpOPkb1XsAoESkDyoHbVHV7sGI1tdDwT52OwJh6L6j9IFT1M+Czw/Y9cpSy51dYfx94P5ixGWOMOTbrSW2MMcYvSxDGGGP8EvUzW1Uo6tq1q86da42cjDGmMkRknqp29XfM7iCMMcb4ZQnCGGOMX5YgjDHG+GUJwhhjjF+WIIwxxvhlCcIYY4xfliCMMcb4ZQnCGGOMX5YgjDHG+FVnelKLyFZgXTVOkQzk11A4Tqor1wF2LbVVXbmWunIdUL1raaGqfmdcqzMJorpEZO7RupuHkrpyHWDXUlvVlWupK9cBwbsWe8RkjDHGL0sQxhhj/LIEcdDLTgdQQ+rKdYBdS21VV66lrlwHBOla7B2EMcYYv+wOwhhjjF+WIIwxxvhlCcJHRB4XkZ9FJFtEvhCRpk7HVFUi8ncR+dV3PR+KSILTMVWViAwTkcUi4hGRkGuSKCJ9RWSZiKwUkZFOx1MdIvK6iOSJyCKnY6kOEWkuIt+KyFLff1t3Ox1TVYlIlIjMFpGFvmv5S42e395BeIlIA1Xd7Vu/C2ivqrc5HFaViMjFwDeqWiYifwVQ1QcdDqtKRKQd4AFeAu5X1ZCZV1ZEwoDlQB8gB5gDXK2qSxwNrIpEpBewBxirqqc6HU9ViUgToImqzheReGAeMCgU/72IiACxqrpHRNzA98DdqjqzJs5vdxA++5ODTywQsplTVb9Q1TLf5kwg3cl4qkNVl6rqMqfjqKJuwEpVXa2qJcAEYKDDMVWZqs4AtjsdR3Wp6mZVne9bLwCWAs2cjapq1GuPb9PtW2rsu8sSRAUi8qSIbAB+AzzidDw15Ebgc6eDqKeaARsqbOcQol9EdZWItATOAGY5G0nViUiYiGQDecCXqlpj11KvEoSIfCUii/wsAwFU9WFVbQ6MB+5wNtpjO961+Mo8DJThvZ5aK5BrCVHiZ1/I3pnWNSISB7wP3HPYE4SQoqrlqtoJ75OCbiJSY4//wmvqRKFAVS8KsOjbwKfAo0EMp1qOdy0icj3QH7hQa/mLpkr8ewk1OUDzCtvpwCaHYjEV+J7Xvw+MV9UPnI6nJqjqThGZDvQFaqQhQb26gzgWEcmosDkA+NWpWKpLRPoCDwIDVLXQ6XjqsTlAhoi0EpEI4CpgssMx1Xu+F7uvAUtV9V9Ox1MdIpKyv5WiiEQDF1GD313WislHRN4H2uJtMbMOuE1VNzobVdWIyEogEtjm2zUzhFtkDQaeA1KAnUC2ql7ibFSBE5HLgGeAMOB1VX3S4ZCqTETeAc7HO7R0LvCoqr7maFBVICI9ge+AX/D+/w7wR1X9zLmoqkZETgfexPvflwuYpKqjauz8liCMMcb4Y4+YjDHG+GUJwhhjjF+WIIwxxvhlCcIYY4xfliCMMcb4ZQnCmEoQkT3HL3XM+u+JSGvfepyIvCQiq3wjcc4QkbNEJMK3Xq86spraxxKEMSeIiHQAwlR1tW/Xq3gHv8tQ1Q7ADUCyb2C/r4ErHQnUGB9LEMZUgXj93Tdm1C8icqVvv0tE/p/vjmCKiHwmIkN91X4DfOwr1wY4C/iTqnoAfKO+fuor+5GvvDGOsVtYY6pmCNAJ6Ii3Z/EcEZkB9ABaAqcBqXiHkn7dV6cH8I5vvQPeXuHlRzn/IuDMoERuTIDsDsKYqukJvOMbSTMX+B/eL/SewLuq6lHVLcC3Feo0AbYGcnJf4ijxTWhjjCMsQRhTNf6G8j7WfoAiIMq3vhjoKCLH+n8wEiiuQmzG1AhLEMZUzQzgSt9kLSlAL2A23ikfs3zvItLwDm6331LgZABVXQXMBf7iG10UEcnYPweGiCQBW1W19ERdkDGHswRhTNV8CPwMLAS+AR7wPVJ6H+88EIvwzqM9C9jlq/MphyaMm4HGwEoR+QV4hYPzRfQGQm50UVO32GiuxtQwEYnzTSKfhPeuooeqbvGN1/+tb/toL6f3n+MD4KEQno/b1AHWismYmjfFN4lLBPC4784CVS0SkUfxzku9/miVfZMLfWTJwTjN7iCMMcb4Ze8gjDHG+GUJwhhjjF+WIIwxxvhlCcIYY4xfliCMMcb49f8B+Z45MDyE9wsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘制交叉验证误差曲线 - log 似然损失(负 log 损失)\n",
    "test_means = grid_neg_log.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_neg_log.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_neg_log.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_neg_log.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penalties = len(penalties)\n",
    "test_scores = np.array(test_means).reshape(n_Cs, number_penalties)\n",
    "train_scores = np.array(train_means).reshape(n_Cs, number_penalties)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs, number_penalties)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs, number_penalties)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penalties):\n",
    "    plt.errorbar(x_axis, -test_scores[:, i], yerr=test_stds[:, i], label=penalties[i] + ' Test')\n",
    "    print(penalties[i].capitalize() + ' Test')\n",
    "    for j in range(0, n_Cs):\n",
    "        print('C={} \\t - Logloss={} '.format(Cs[j], -test_scores[j,i]))\n",
    "\n",
    "for i, value in enumerate(penalties):\n",
    "    plt.errorbar(x_axis,-train_scores[:, i], yerr=train_stds[:, i], label=penalties[i] + '- Train')\n",
    "    print('\\n' + penalties[i].capitalize() + ' Train')\n",
    "    for j in range(0, n_Cs):\n",
    "        print('C={} \\t - Logloss={} '.format(Cs[j], -train_scores[j,i]))\n",
    "    \n",
    "plt.legend()\n",
    "plt.xlabel(\"log(C)\")                                                                                                      \n",
    "plt.ylabel(\"neg logloss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由上面数据列表与曲线图可知: 在给定 L1 正则和 L2 正则的时候、不同正则参数 C 对应的模型在训练集与测试集上的负 log 损失。  \n",
    "在训练集上 C 越大的模型性能越好, 但在测试集上当 C=1 时性能最好。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 绘制交叉验证误差曲线 - 正确率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L1 Test\n",
      "C=0.001 \t - Accuracy=0.6510482981071216 \n",
      "C=0.01 \t - Accuracy=0.7070622188269248 \n",
      "C=0.1 \t - Accuracy=0.7630421865715983 \n",
      "C=1 \t - Accuracy=0.7721670486376369 \n",
      "C=10 \t - Accuracy=0.7708683473389355 \n",
      "C=100 \t - Accuracy=0.7708683473389355 \n",
      "C=1000 \t - Accuracy=0.7708683473389355 \n",
      "L2 Test\n",
      "C=0.001 \t - Accuracy=0.7474492827434004 \n",
      "C=0.01 \t - Accuracy=0.7617859264918089 \n",
      "C=0.1 \t - Accuracy=0.7747644512350395 \n",
      "C=1 \t - Accuracy=0.7708683473389355 \n",
      "C=10 \t - Accuracy=0.7708683473389355 \n",
      "C=100 \t - Accuracy=0.7708683473389355 \n",
      "C=1000 \t - Accuracy=0.7708683473389355 \n",
      "\n",
      "L1 Train\n",
      "C=0.001 \t - Accuracy=0.6510420804533779 \n",
      "C=0.01 \t - Accuracy=0.7070331823839411 \n",
      "C=0.1 \t - Accuracy=0.7698583194300997 \n",
      "C=1 \t - Accuracy=0.7747395460925294 \n",
      "C=10 \t - Accuracy=0.7747406053865099 \n",
      "C=100 \t - Accuracy=0.7744148724874871 \n",
      "C=1000 \t - Accuracy=0.7744148724874871 \n",
      "\n",
      "L2 Train\n",
      "C=0.001 \t - Accuracy=0.7467492915971505 \n",
      "C=0.01 \t - Accuracy=0.7640004237175922 \n",
      "C=0.1 \t - Accuracy=0.7731135298323667 \n",
      "C=1 \t - Accuracy=0.7737634066894415 \n",
      "C=10 \t - Accuracy=0.7744148724874871 \n",
      "C=100 \t - Accuracy=0.7744148724874871 \n",
      "C=1000 \t - Accuracy=0.7744148724874871 \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'accuracy')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xUVdrA8d8zk04CKYQaSkAQpEsMqFjAxoKK6KLgqqurou+q6+rqWlcRC9a1rK6ioKi4IuLaEAXEAroCoUkVaSKhExJCSEKSmef9YyYxhAGGJMNNeb4fx8w9c86d5waYZ+45554rqooxxhhTkcvpAIwxxtRMliCMMcYEZAnCGGNMQJYgjDHGBGQJwhhjTECWIIwxxgQU0gQhIgNFZLWIrBWRuwO83lpEvhaRxSKyVEQGlXvtHn+71SJyXijjNMYYczAJ1XUQIuIGfgbOATKBDGCEqq4sV+dVYLGqviwiJwDTVLWt//m7QDrQAvgS6KiqnpAEa4wx5iChPINIB9aq6npVLQImAUMq1FGgof95I2CL//kQYJKq7lfVDcBa//6MMcYcI2Eh3HdLYFO57UygT4U6o4AZInIL0AA4u1zbuRXatjzcmzVu3Fjbtm1bhXCNMab+Wbhw4S5VTQ70WigThAQoq9ifNQKYoKrPiMjJwNsi0jXItojISGAkQOvWrVmwYEEVQzbGmPpFRDYe6rVQdjFlAq3KbafwWxdSqWuByQCq+gMQBTQOsi2q+qqqpqlqWnJywARojDGmkkKZIDKADiKSKiIRwHDgkwp1fgXOAhCRzvgSxE5/veEiEikiqUAHYH4IYzXGGFNByLqYVLVERG4GpgNu4HVVXSEio4EFqvoJ8DfgNRG5DV8X0tXqm1a1QkQmAyuBEuAmm8FkjDHHVsimuR5raWlpamMQxhhzdERkoaqmBXrNrqQ2xhgTkCUIY4wxAVmCMMYYE5AlCGOMMQFZgqhL3hjsexhjTDWwBGFqJkt2xjjOEoQxxpiALEEYY4wJyBKEMaFWh7rLVjzWjxWP9XM6jCqrK8cBoT2WUK7maowBVmzdA0AXh+Mw5mjZGYQxxpiALEEYY4wJyBKEMcaYgCxBGGOMCcgShDHGmIAsQRhjjAnIEoQxxpiALEEYY4wJyBKEMcaYgEKaIERkoIisFpG1InJ3gNefFZEl/sfPIpJT7rUnRWSFiKwSkRdEREIZqzHGmAOFbKkNEXEDLwHnAJlAhoh8oqorS+uo6m3l6t8C9PI/PwU4Fejuf/k74Azgm1DFa4wx5kChPINIB9aq6npVLQImAUMOU38E8K7/uQJRQAQQCYQD20MYqzHGmApCuVhfS2BTue1MoE+giiLSBkgFvgJQ1R9E5GtgKyDAi6q6KkC7kcBIgNatW1dr8MZUl4ca5wEw2eE4jDlaoTyDCDRmoIeoOxyYoqoeABE5DugMpOBLNANE5PSDdqb6qqqmqWpacnJyNYVtaoKN/9nCxv9scToMY+q1UCaITKBVue0U4FD/4ofzW/cSwFBgrqrmqWoe8DnQNyRRGmOCtvGr3Wz8arfTYVRZXTkOCO2xhDJBZAAdRCRVRCLwJYFPKlYSkeOBBOCHcsW/AmeISJiIhOMboD6oi8nUXfuKSthXVOJ0GNXi6slFXD25yOkwjDlqIUsQqloC3AxMx/fhPllVV4jIaBG5sFzVEcAkVS3f/TQFWAcsA34EflTVT0MVqzHGmIOF9I5yqjoNmFah7IEK26MCtPMAN4QyNmOMMYdntxw1NdIv4SWEuZUTVKGWXyNZ4oJiN+QW5TodSpUVhPt+5hfnIyK4xOV74Ptp17PWLZYgTM2hCpkZsGwKZ6VvIzLCC4+1gLjm0LD0Z3OIa/Hbz7hmvoc73LGwveplV8EuMvdmsjlvM5l7M8nMyyz7ueMPbgBGvXtq5d5AFbcXXF4Iq/jTc4jystf10O30t/bugx7623NPufI4X0hv/CHtkOH6koRQ+n8R8W8R8Lkg+P6TsgRT9kr57UPtp/xr8ltbEMS/Xb5cEPLDfD3aU6/7XeX+TGqQfWFKUXhoErMlCOO87Sth+RRYNgVyNkJYFLv3RLInL5xOI66EvVsgdytsmgt7t4HntwFfVUAFjU5GY5tCTDM0pgka3QRiktGoJDQ6CY1MAHc06vGgJR60pBg8HrSkpOyBx4MWl6CeEigp8dcrAU8J+/fnsyd/N7n52eTm55BXkMO+wlzyC/dSsD8PSjxlH6JhXugikaS7oomRSHRLCWEeiG3eCvF4yx54vIjHg3gUKfEgXi9S4q3w04N4DzU7PLS8YS7U7UbdLtTtAreLooJ9AITFxpX+Cfz2/7Iwy5VxiDJ/A0XLlQZop0Hsq3RLy5cdvl2ix1ciuzdR2yV4YUej0PwdsQRhnJG9EZZ/4EsKO1aAuNG2Z1LY8kryd8WwZcqLhJXA2q0L/R/WJainEVoS49suLkY9HvB4y+00y/9YEbKw4/yPEhd43YK63UiYGwmLxBUeQVhYBGERkUhYOBIWhrjd5OzZBgKJsc2RMDeEhSHuMCQsDMLcvrpuNxIeBu6K2756gdv5t8u3C3Mf8FzCwgJs+9qI27/P0ofbDeH+93a7Ax7/tLNPAGDQl/ND9js+Fn47juUOR1J1084+gcgQ7dsShDl28nbCyo9g2fuwaR7qgYKwXuSXDCM/s4j8j1eg+b7ZzGFhUBQOjXv0qNyHp4AU70VK8qBoD+zfQ8H+LLIKd7GrOIcdnr3sYD/bw4Vt4W62h7spcgseF3hcgEBSWDjNImJpGp1I84bNaZGQSqvGnWjZvAeNGrZGXIefBFhU4mVpZg4T77+fQlckXa/+M1HhbiLD3USFucp+RoW7/Q//8zDf80h/WYTb+vaNMyxBmNAqzIWfPoNl7+Nd8w0Fu9zk70shf08aBRuy0P3bge1EduhA/EUXEZN+EjFpacwY7rtw/qSnngz6rYo8RWzJ21JuHCDvgHGBvcV7y9V2kRDZgpSYZqRExtPFHUOKukgp8dCycB/N8rIJy9sGuzOhYOnBbxYe4x//KB0PaU5xg2ZsKGrI4uwYvtseztdbhLxiF6QMxqUevpjxc6V+hSKUJY3SZBJZllgOTCplr4e7/GXly38riwzQLrJcsnK7LCEZSxAmFIoLYe1MvAsnUfDDN+RvE/J3x1Gwszla4gXZT2SnRsRfdhYxJ/kSQlhCwhF3q6pkFWYdOAhcmgDyMtm+b7u/T9snwhVBy7iWpMSm0CO5BylxKb5HrO9ng/AGwR/P3q2+R+4W3zjI3q2U5Gxm365N6LbvabB/J+EU0xHoCFwGqFsoikliX9Ze8Cjx3Xrj9SpeBa8q6v/pVcXrLfe8XHlZnfKve8BbongLfOV6ULvffh72sPyPvQFecwEu8Q3yulyCS3zbKenhgLLy0UoOuNcQKem+j77afhzgO5ad++OOXLESLEEAvDHY9/Oaz5yNozbzevCsmEHB52+Sn5FB/lYo2B0O3obgEqJO6ELCwJN8CaH3ibgbNTrs7nKjYWkbYdn8J8oSwua8zRSUFBxQr0l0E1LiUkhvlk7L2JYHJIDG0Y1xSTVcCxoeBYmp7GvQioX7s5m7OYt5G3azNDOHYo/iEujaoiFntnJzSpMiusXl06BoB5K7lci9W9jz5Xu4XOByhXGEXqlqpXBAsjkgyQRKSN5A2xXq+WcHqQQeo6gtvP5rhGv7cYDvWDTg0ndVZwmiDild3K7NNcfm/Tx79pA/833yZ31E/vK1FO4CVMAVSXTHtiQN6U9Mnz5En3gi7tjYoPdb4i3h9QEutiYKMWv+S0pcCq3jWnNKi1N+SwJxKbRo0IKosKiQHd/ewmIW/JLN3A1ZzFu/m2Wb9+DxKm6X0K1lI/7UL5W+qUn0bptAw6hDT7Nd8MIsAAY9NzVksQYigNv/qC4rHusHQJd7Z1fjXo+9unIc4DuW5sH/8zoqliBM0Dw5OeQvWED+tzPYN/c79mdmg4K4lKiUOJIuOYmYc4YRc1IfXDExlX6f939+n62Jwog5Xu55be4xG6Ddk1/M/F92M2+97wxhxZY9eBXC3UKPlHhuPKMdfVKT6N0mgQaRwf/TaTMgMYRRGxM6liDMIZVkZZGfscDXZTT3f+xftwEAcSvRSUU07teCmDN+R/T51+GKb1ot75ldmM2Li1+k/Valxy8a0uSwe18R8zdkMXf9buZt2M1P23JRhYgwF71axXPzgA70TU2kV+sEoiNqf1eEMUfLEoQpU7xjhy8ZZGSQn7GAonXrAJBwISaxgIbdiojp0o6os4fj6jkMYptUewz/Wvwv9hXv49oMb7X3qu7cu5/5G3Yzd30W8zZk8fN23418osJd9G6TwG1nd6RPaiI9WsUTFW4JwRhLEPVY8datvyWE+RkUbdwIgCs6gugWETTqsZcGyYVEdWyH9PwTdL0EElNDFs/KrJVM+XkKf+j8B5q99WaV97c9t9CfDHzdRut2+q4Cjolw07tNAkN6tqRPaiLdU+KJCDuGo8fG1BKWIOoJVaV482by52eUJYXizEwAXA3jiDm+FfGdWxMjy4mKy0MSWkO366Dr76Fpl5AvmKeqjJk3hoSoBP7c889ce6nv/lGDjmIfm3MKfOMH63czb0MWv2TlAxAXGUZa2wSGpbWiT2oiXVs2ItxtCcGYI7EEUUepKsUbN7KvXJdRydatALjj44lJ603ioJOJid5IZPbXSNFqiGkMXUZAt99DSjrHck7m1PVTWbJzCaNPGU1cxJHndKsqm3YXlM0wmrchi8xs3xTYhlFhpKcmcUXfNvRJTeKEFg3twi9jKsESRB2iHqWkwMPm2/9GfkYGJTt3AuBOSvJdf3DttcS0bUBk7lxk5YeQuw0KY6HzBb4zhXZnOLIqal5RHv9c+E+6Ne7GkOOGBKyjqmzYta+su2jeht1s3VMIQEJMOOmpifzp1FT6tEukUzNLCMZUB0sQdUDx1q3snvAmBVv3+1a0XLCAmPR0X1JIP4mI2GJk+RRY/jSsXQ/uCOhwru9MoeNACI92NP6xS8eyq2AXL/R/oezCNlUoLE5m4tyNZUlhx979ADSOjaBPahJ92iXSJzWJDk1icVlCMKbaWYKoxfavXUvWuPHsmTrVd8+AaBfhcWG0/eYbJHezb7XUGX+EbUtBXND2NOh3u++MITre6fABWL9nPRNXTmTocUPpltytrDwhcxgzinuyLHM5TRtG0rfdbwmhfXIDW7zOmGMgpAlCRAYCz+O7mHOcqj5e4fVngf7+zRigiarG+19rDYwDWuFbNWCQqv4Synhri/xFi8kaN468r75CoqJIGD6cxKuvZvt1vyOmcR4yYTBs/N5XuWVvGPg4dBnqW1yuBlFVnpj/BNFh0dx64q1l5TNWbGNGcU8GhS/i77feTpukGEsIxjggZAlCRNzAS8A5QCaQISKfqOrK0jqqelu5+rcAvcrt4i3gUVWdKSKxQPmF/+sdVSXv22/Jem0cBQsX4m7UiMY33UTC5SMI27Mc5j1Iy5N+xeVW2JcI/e+HrhdDUnunQz+krzZ9xf+2/I+7TrqLpOgkAHLyi7jvo+WkurYzMmombRv/w+Eojam/QnkGkQ6sVdX1ACIyCRgCrDxE/RHAg/66JwBhqjoTQFXzQhhnjabFxeROm0bWuPHsX7OGsObNaXrvvcSfezKu1R/CxAGQ/QtENSJvRxz7tsfR/MP5Nf4+zoUlhTyV8RTHxR/HZZ0uKysfPXUlu/cV8Y/ozwiXev2dwBjHhTJBtATK388vE+gTqKKItAFSga/8RR2BHBH5r7/8S+BuVfWELtyaxZufT86UD8ia8AYlW7YS2aEDLR57hIbHCbL0PzD2L6BeSD3dd7bQ+XyyB53ia1zDkwPAGyveYHPeZsafO55wl2/m1Fc/bee/izZzy4DjaL9gu8MRGmNCmSACfUodaoX64cCUcgkgDDgNX5fTr8B7wNXA+APeQGQkMBKgdevWVY+4BijJziZ74jtkv/MOnpwcotN60+zWa4mNXIEs/Tss3eW7SU2/26HXFSG9sjlUtuRtYfyy8Zzb5lzSm6cDsKegmHv+u4yOTWO5ecBxrF3gcJDGmJAmiEx8A8ylUoAth6g7HLipQtvF5bqnPgL6UiFBqOqrwKsAaWlpztzZvZoUb95M1hsTyPngA7SggNgzTyfpzDbE5H0Ni/8MrjA4/nfQ6yo47ixw1d61gp5e8DSCcEfaHWVlj362kl15Rbx2VRqRYbX32IypS0KZIDKADiKSCmzGlwQur1hJRI4HEoAfKrRNEJFkVd0JDADq5HfKwtU/kzV+HLmfTQMRGvXvQ1K3EiJ3fgo/7YPGx8O5j0D34RCb7HS4VTZ361xmbpzJzT1vpnlscwC+Wb2DyQsy+b8z29M9pWZMvzXGhDBBqGqJiNwMTMc3zfV1VV0hIqOBBar6ib/qCGCSqmq5th4RuQOYJb75jQuB10IV67GmqhQsXMiu115j37ezkehoEgd0IbHFOsL3vw87G/hmIJ14FaScFPSYQpvLW4Q48qop9hYzZt4YUmJTuLrr1YDvpjz3/HcZxzWJ5dazOjgboDHmACG9DkJVpwHTKpQ9UGF71CHazgS6hyw4B6jXS97XX/umqi5ZgrthA5L7NychcSnu8HXQpA/0etF3zUJkiG4R5aB3V73L+j3reaH/C0S6IwF4bNpPbM8t5IP/O8WW2DamhrErqY8BLSpiz9TPyBo/nqJ16whPakDTUyC+xVpcDZOgxw2+s4Xk450ONWR2Fezi5R9f5tSWp3JmqzMB+G7NLt6d/ysjT29Hr9YJB9QfnfQU4JudYIxxhiWIEPLk7SPn/ffZPWECJdu3E9kkghYnZ9Ow9Tak49nQ6wnfWkhhEU6HGnLPLXyOQk8hd590NyJC3v4S7vpgKe0aN+D2czo6HZ4xJgBLEMDG//gmV7W5pnr2V5KVxe633yZ74kS8efuIaeah+Rk5NOjcDDnxb9DjcmjUsnrerBb4ceePfLzuY67peg1tG7UF4PHPV7FlTwFTbjzZupaMqaEsQVSjok2b2P3aK+R8+DFa7CEupYCkfkVEnzYYTrwS2vQ7pvdYqAm86mXMvDEkRydzQ/cbAPjful1MnPsr1/ZLpXebRIcjDL0uzRs5HYIxlWIJohoUrlhB1guPkzt7AYjSqG0+SaelEHnW7b4ltaMTjryTOuqjtR+xImsFY04bQ4PwBuQX+bqW2ibFcMe5dXfMxZi6wBJEJakq+V9NI+ulZ9m3cjOuMC+JXUpIvGQQ4WdeD83r1ASsSsktyuX5Rc9zYpMTGZw6GIAnv1jNpt0FvDeyL9ER1rVkTE1mCeIo6f4C9r7zT7ImfkDhlgLcUR6Sz0gm4eobcKcNc/zmOzXJv5f8m5z9OdzT5x5EhHnrs5jwv1+4+pS29GmX5HR4xpgjsAQRJG/mMvaMG8PuaYsoyhXCG0KzEek0GnkvrubWVVLRmuw1TPppEr/v8Hs6JXaioMjD3z9YSqvEaP4+sJ79vq75zOkIqk2Xe79zOoRqUVeOA0J7LJYgDmf/Xjzz3yXn7fHsnp9NSaGbqOYNaHnjpcRdeRsSXvenp1aGqjJm/hhiI2K5pdctADw9YzUbs/L5z/V9iImwv3bG1Ab2L7UiVdg0j5Jvx7H7o1lk/xyBt9hFTOc2tLjlb8T0P8/ubnYE0zdOJ2NbBvf3uZ/4qHgWbtzN699v4Iq+rTmlfWOnwzPGBMkShJ8rvAS+f56ir98i64ed7NkQg2oUcaelk/SXO4nu2tXpEGuF/OJ8ns54mk6Jnfh9x99TWOzhzveX0qJRNHf/rrPT4RljjoIliNwtJHfahmgxmaOeZe+maCS8IY0uHkLS9SOJaNPG6QhrlXHLxrE9fztPnv4kbpebJ79Yxfpd+5h4bR9iI4P/6/beDSeHMEpjTDDqfYIo3rOfHYtjKNgVgatBDEnX/4HEq64kLLn2L619rG3K3cSEFRMY3G4wJzY9kcW/ZvPanPWMSG9Fvw7WtWRMbVPvE4S7SQsK90QS3shN6qxvccfWvVVUj5UnM54k3BXO7b1v93UtTVlKs4ZR3DvIupaMqY3q17oPAbgiI4lqGkF4wzBLDlUwJ3MO32R+ww09bqBJTBNemLWGtTvyGHNJd+Kiwp0OzxhTCfU+QQA2K6mKijxFPJHxBG0btuXKzleyNDOHsbPXM6x3Cmd0tK46Y2qret/FZKru7ZVvszF3Iy+f/TJedXHn+0tpHBvB/eef4HRoxpgqsDMIUyXb921n7NKxnNnqTPq17MeLX61l9fa9jLm4G42irWvJmNospAlCRAaKyGoRWSsidwd4/VkRWeJ//CwiORVebygim0XkxVDGaSrvnwv/icfr4e8n/Z3lm/fw72/WcfGJLRnQqanToRljqihkXUwi4gZeAs4BMoEMEflEVVeW1lHV28rVvwXoVWE3DwPfhipGUzULty9k2oZpjOw+kqbRLRn5+vckNojgAetaMqZOCOUZRDqwVlXXq2oRMAkYcpj6I4B3SzdEpDfQFJgRwhhNJXm8HsbMG0OzBs24rtt1/PubtazamsujF3UlPsbWqDKmLghlgmgJbCq3nekvO4iItAFSga/82y7gGeDOEMZnquD9n99ndfZq7ki7gw07innxq7UM6dmCc7s0czo0Y0w1CWWCCDR3VA9RdzgwRVU9/u0/A9NUddMh6vveQGSkiCwQkQU7d+6sQqjmaGQXZvOvxf8ivVk6/VPO5s4pPxIfE86oC7o4HZoxphqFcpprJtCq3HYKsOUQdYcDN5XbPhk4TUT+DMQCESKSp6oHDHSr6qvAqwBpaWmHSj6mmr24+EX2Fe/j7vS7eXX2elZsyeWVK04koYF1LRlTl4QyQWQAHUQkFdiMLwlcXrGSiBwPJAA/lJap6h/KvX41kFYxOVSnNpe3CNWu65xVWat4/+f3ubzz5Xj3N+P5WXMY3L05A7s2dzo0Y0w1C1kXk6qWADcD04FVwGRVXSEio0XkwnJVRwCTVNXOAGq40hsBJUQlcEO3G7lzyo/ERYUz+kLrWjKmLgrpldSqOg2YVqHsgQrbo46wjwnAhGoOzVTC1PVTWbxjMQ+d8hCT5mWxNHMPL17ei6TYSKdDM8aEgF1JbYKyr3gfzy58lq5JXena8CyenfkzA7s0Y3A361oypq4KKkGIyAciMtg//dTUQ2N/HMvOgp3cddLd3PXBcmIi3Tx8UVdb6NCYOizYD/yX8Q0wrxGRx0WkUwhjMjXMhj0beHvV21x03EVkrI5j8a85PHRhF5LjrGvJmLosqAShql/6ZxadCPwCzBSR/4nINSJiK7LVYarKE/OfIModxdA21/P0jNWcc0JTLuxhM7+MqeuC7jISkSTgauA6YDHwPL6EMTMkkZka4etNX/P9lu+5scf/8dinmUSFu3nUupaMqReCmsUkIv8FOgFvAxeo6lb/S++JyIJQBWecVVhSyJMZT9K+UXv27z6ZBRt/5plhPWjSMMrp0Iwxx0Cw01xfVNWvAr2gqmnVGI+pQSasmMDmvM080udF7npnLQM6NeHiEwMup2WMqYOC7WLqLCLxpRsikuBfBsPUUVvytjB+2XjOaX0O73wbQbjbxWNDu1nXkjH1SLAJ4npVLbuZj6pmA9eHJiRTEzy94GkAUl3Dmb9hN/84/wSaNbKuJWPqk2C7mFwiIqXLYfhvBmQrs9U013xWLbuZu3UuMzfO5IqOI3n5892c3jGZYb1TqmXfxpjaI9gEMR2YLCKv4Fuy+0bgi5BFZRxT7C3m8XmP0zK2JYuW9cQl+Tx+sXUtGVMfBdvFdBe+m/n8H75luWcBfw9VUKZyLhv7A5eN/eHIFQ9j0k+TWLdnHX0aXsO89bncN7gzLeKjqylCY0xtEtQZhKp68V1N/XJowzFO2lWwi38v+Te9k/syZU5D+h2XyPCTWh25oTGmTgr2OogOwBjgBKBspFJV24UoLuOA5xc9T6GnkH1bzweEMda1ZEy9FmwX0xv4zh5KgP7AW/gumjN1xNKdS/lo7UekxQ8hY00Ydw/qTKvEGKfDMsY4KNgEEa2qswBR1Y3+ezgMCF1Y5ljyqpcx88aQGJnEDwt70bddIn9Ib+10WMYYhwU7i6nQv9T3GhG5Gd8tRJuELixzLH289mOWZy2njfdadngiefKSHrhc1rVkTH0XbIL4KxAD/AV4GF830x9DFdQxV03XD9RGuUW5PLfoOVKiO7N80XGMuuB4WidZ15IxJogE4b8o7lJVvRPIA64JeVTmmHl5yctkF2aze/NVpLdN4qqT2zodkjGmhjjiGISqeoDeUonpLCIyUERWi8haEbk7wOvPisgS/+NnEcnxl/cUkR9EZIWILBWRy472vc2Rrclew7s/vUtjPYPighY8+fvu1rVkjCkTbBfTYuBjEXkf2FdaqKr/PVQD/5nHS8A5QCaQISKfqOrKcu1vK1f/FqCXfzMfuEpV14hIC2ChiEwvvx6UqRpV5fH5jxPhimb96tO4f+DxtG3cwOmwjDE1SLAJIhHI4sCZSwocMkEA6cBaVV0PICKTgCHAykPUHwE8CKCqP5e9ieoWEdkBJAOWIKrJjI0zmL9tPpJ1Mb1TUrjm1FSnQzLG1DDBXkldmXGHlsCmctuZQJ9AFUWkDZCKbzmPiq+l41sYcF0lYjAB5Bfn83TG08TQipzd6Tx5a3fc1rVkjKkg2Cup38B3xnAAVf3T4ZoFKDtoH37DgSn+8Y7y79sc3wV5f/Qv91ExrpHASIDWrW3efrDGLx/Ptvxt5P9yA3ed24n2ybFOh2SMqYGC7WKaWu55FDAU2HKENplA+YV8Ug7TZji+RQDLiEhD4DPgflWdG6iRqr4KvAqQlpZ2qORjytm0dxNvLJ+A7OtFt8a9uLafrZZijAks2C6mD8pvi8i7wJdHaJYBdBCRVHwX1g0HLq9YSUSOBxKAH8qVRQAfAm+p6vvBxGiC82TGk3i8QuH2QTz1Z+taMsYcWrBLbVTUAThsn46qlgA347uXxCpgsqquEJHRInJhuaojgEmlNyPyuxQ4Hbi63DTYnpWM1fh9t/k7vtn0DQU7+vPX/ml0aBrndEjGmBos2DGIvRw4frAN3z0iDktVpwHTKpQ9UGF7VIB2E0NIVXwAABnWSURBVIGJwcRmglPsKeaxuWOgOJnjowcx8jTrWjLGHF6wXUz2VbOWe3vV22zK+5Wi7dfw9LVphLkre/JojKkvgvqUEJGhItKo3Ha8iFwUurBMddqRv4N/L36Fkr2dubnvBRzfzPK9MebIgv0a+aCq7ind8F/R/GBoQjLV7fF5T7PfU0wrhnPjme2dDscYU0sEmyAC1Qt2iqxx0KLti5j56+cU7z6d5y4+m3DrWjLGBCnYT4sFIvJPEWkvIu1E5FlgYSgDM1Xn8Xq4d/ZovMWN+FOXazmhRUOnQzLG1CLBJohbgCLgPWAyUECFC9uM836JeJpfIp4u2357xXtszl9H0v5L+OtZXR2MzBhTGwU7i2kfcNBy3abmyinM4YVF/8KT346XhlxDRJh1LRljjk6ws5hmikh8ue0EEZkeurBMVd399ZMUaT4Xt76JbinxR25gjDEVBPu1snH5ezGoajZ2T+oaa8HWZXy/fSoN9p/GP847y+lwjDG1VLAJwisiZUtriEhbDr0yq3GQotz+5UOoJ4Znz7mLyDC30yEZY2qpYKeq3gd8JyLf+rdPx7/MtqlZCveHUeBaRd+EGzmlXasjNzDGmEMIdpD6CxFJw5cUlgAf45vJZGoQj8eFJ3wTESWteeF8y9/GmKoJdrG+64Bb8d3TYQnQF9/y3AMO184cWwX7BVdsHvec+BQxEeFOh2OMqeWCHYO4FTgJ2Kiq/YFewM6QRWWO2oSFs3DFrsK773gu7dbP6XCMMXVAsAmiUFULAUQkUlV/Ao4PXVjmaGTt28uzSx7BWxxPTKTT0Rhj6opgB6kz/ddBfATMFJFsjnzLUXOMXPvpI3jDdhFW0BVXZJHT4Rhj6ohgB6mH+p+OEpGvgUbAFyGLygTt7cXfsLbwc46LOoet3o1Oh2OMqUOOev0FVf1WVT9RVfuq6rDd+Xk8vehh3N4Exl0wyulwjDF1TEgX6BGRgSKyWkTWishBazmJyLPl7jn9s4jklHvtjyKyxv/4YyjjrK2u//QxvGE7uLXHvTRuYCu1GmOqV8ju6SAibuAl4BwgE8gQkU9UdWVpHVW9rVz9W/DNjkJEEvHdkCgN3xXbC/1ts0MVb20zael3rC6YSmrkAP7U+zynwzHG1EGhvOlPOrBWVdcDiMgkYAiw8hD1R/DbXerOA2aq6m5/25nAQODdEMZba+QU5PN4xkO4pBHjL3iorLxt0R0ORmWMqWtC2cXUEthUbjvTX3YQEWkDpAJfHW3b+mjkp2PwhG3jlm730CTWVmo1xoRGKBOEBCg71AJ/w4Epquo5mrYiMlJEFojIgp0768d1e1OW/cDK/E9oFX4G1580yOlwjDF1WCgTRCZQfrW4FA597cRwDuw+Cqqtqr6qqmmqmpacnFzFcGu+3MICHpn/IC5vHOMveNjpcIwxdVwoE0QG0EFEUkUkAl8S+KRiJRE5HkjAt7ZTqenAuf4bEyUA5/rL6rUbPn0CT9hWbuxyF83jEpwOxxhTx4VskFpVS0TkZnwf7G7gdVVdISKjgQWqWposRgCTVFXLtd0tIg/jSzIAo0sHrOurj1fOZ9m+D0kJ78ef+1zgdDjGmHoglLOYUNVpwLQKZQ9U2B51iLavA6+HLLhaJG9/IaN+eACXxDLufOtaMsYcG3Yn+1rgxqlPURK2mWs730FKo8ZOh2OMqScsQdRwn/20kCV7p9DMfQq3njz0yA2MMaaahLSLyVTNvqL9/OP7fyASw/jBjzgdjjGmnrEziBrsz1OfoThsE1d3vJ3WCXV/Gq8xpmaxBFFDffHzEhbmTqaJK52/9RvmdDjGmHrIEkQNVFBcxL1z7kO8UYwb/JjT4Rhj6ilLEDXQzVOfpTjsV6447jZSE5s6HY4xpp6yQeoaZtbaZczLmUSyO42/n3ap0+EYUysUFxeTmZlJYWGh06HUWFFRUaSkpBAeHh50G0sQNcj+4mLumn0fQgSvDnoUkUBrFhpjKsrMzCQuLo62bdvav5sAVJWsrCwyMzNJTU0Nup11MdUgt3z+AvvdGxje7i90aNzC6XCMqTUKCwtJSkqy5HAIIkJSUtJRn2FZgqghvt2wgv9lTSSRXtx7xuVOh2NMrXO0yeGysT9w2dgfjlyxjqhM8rQEUQMUlZRwx9f3IRrO2N+NsW9BxtRCsbGxZc8HDhxIfHw8559/fsC6N910Ez179uSEE04gOjqanj170rNnT6ZMmXJU77lo0SK++OKLKsV9ODYGUQP89fMXKXSv45JWd9Cpid04z5ja7s477yQ/P5+xY8cGfP2ll14C4JdffuH8889nyZIllXqfRYsWsXz5cgYOHFjpWA/HziAc9v0vq5i96y3i6c4DZ17pdDjGmGpw1llnERcXV6m2a9as4bzzzqN3796cfvrp/PzzzwBMmjSJrl270qNHD/r3709BQQGjR4/mnXfeqdTZRzDsDMJBxR4Pt311HyJuXhk4BpfL8rUxVfXQpytYuSX3iPVWbvXVCWYc4oQWDXnwgi5Vji0YI0eOZNy4cbRv357vv/+em2++mRkzZvDQQw/xzTff0LRpU3JycoiOjuaBBx5g+fLlPPfccyGJxRKEg/72xcsUuNcwpOVf6dK0dZX3994NJ1dDVMYYp+Tk5DB37lwuueSSsrKSkhIATj31VK666iqGDRvGxRdffEzisQThkLm/ruGrHW/QSLoyesA1TodjTJ0R7Df90jOHmvTFSlVp3LhxwDGJ1157jXnz5jF16lR69OjB0qVLQx6P9Wk4wOPxcuuX9yIIL59nXUvGGJ+EhASaN2/Ohx9+CIDX6+XHH38EYP369fTt25eHH36YhIQENm/eTFxcHHv37g1ZPPbJ5IA7Z4wl3/0Tg1uOpHvztk6HY4ypZqeddhrDhg1j1qxZpKSkMH369KDbTpo0iVdeeYUePXrQpUsXpk6dCsBtt91Gt27d6NatG2effTZdu3ZlwIAB/Pjjj/Tq1av2DVKLyEDgecANjFPVxwPUuRQYBSjwo6pe7i9/EhiML4nNBG5VVQ1lvMfCgsx1zNg2njjpzGNnX+d0OMaYapKXl1f2fM6cOUG1adu2LcuXLz+grF27dgETyieffHJQWXJyMgsWLDjKSIMXsgQhIm7gJeAcIBPIEJFPVHVluTodgHuAU1U1W0Sa+MtPAU4FuvurfgecAXwTqniPBa/Xyy0z7wWUl861riVjnFSTxh5qqlB+QqUDa1V1vaoWAZOAIRXqXA+8pKrZAKq6w1+uQBQQAUQC4cD2EMZ6TNw9czx5rpWc1/xaTmzR3ulwjDHmsEKZIFoCm8ptZ/rLyusIdBSR70Vkrr9LClX9Afga2Op/TFfVVRXfQERGisgCEVmwc+fOkBxEdVm8ZQPTNr9KA+/xPHnuDU6HY4wxRxTKBBFoQaGKYwhhQAfgTGAEME5E4kXkOKAzkIIvqQwQkdMP2pnqq6qapqppyck1957NXq+Xm6bfC+LhhbPH4Ha5nQ7JGGOOKJQJIhNoVW47BdgSoM7HqlqsqhuA1fgSxlBgrqrmqWoe8DnQN4SxhtT9s95gr2s5Zze9mvRWHZwOxxhjghLKBJEBdBCRVBGJAIYDFYfhPwL6A4hIY3xdTuuBX4EzRCRMRMLxDVAf1MVUGyzf9iufbhpLjLcDT593k9PhGGNKvTHY9zCHFLIEoaolwM3AdHwf7pNVdYWIjBaRC/3VpgNZIrIS35jDnaqaBUwB1gHLgB/xTX/9NFSxhorX6+XGL+5FpZjnBjxGmNu6loypq0qX+16yZAknn3wyXbp0oXv37rz33nsH1bXlvgFVnQZMq1D2QLnnCtzuf5Sv4wFq/UjuqK8nskd+5MzGf+LkNp2cDscYcwzExMTw1ltv0aFDB7Zs2ULv3r0577zziI+PL6tjy33Xc6t2ZPLhxheJ9rTj2YG3OB2OMeYY6dixIx06+MYaW7RoQZMmTTiaWZa23Hcd5/V6ueFzX9fSM/0fIzzMfs3GHDOf3w3blh253jb/YnfBjEM06wa/O2ghiCOaP38+RUVFtG8f/HVPttx3HffIt++SzWJOa/xHTmt7bNaQN8bULFu3buXKK6/kzTffDHrVBFvuu45bvXML7294gUja8vzv/up0OMbUP8F+0y89c7jms2oPITc3l8GDB/PII4/Qt2/wM/Rtue86TFW5Ydr9qOznqTMeI8K6loypd4qKihg6dGjZt/2jYct912FjZk8miwxOThxB//bdnA7HGOOAyZMnM3v2bCZMmFA2ffVoZinVpOW+pQ6soA1AWlqahnLZ2yNZl7Wdiz6+iAgS+f7KD4kKj3AsFmPqm1WrVtG5c+ejaxTCLqaaKtDvSUQWqmpaoPrWB1JNrv/sPtRVwOP9HrXkYExtUI8SQ2VZF1M1eGrOFHbqPNLjL+Wc43o6HY4xxlQLO4Oool+yd/LWmn8STgovDv6b0+EYY0y1sQRRRddNvR917ePRk18gJjzS6XCMMabaWBdTFTz3/Uds9/6PExv+nt8dH3CMxxhjai07g6ikTTm7GL/6acJpwSvn/93pcIwxR+maL64B4I2BbzgcSc1lZxCVdN3UB1DXXh46ZTQxEda1ZEx9V7rcN8DAgQOJj4/n/PPPD1i3Opb7/vDDD3nqqaeqHPfh2BlEJbw491O2eObQI/ZiLujUx+lwjDE1zJ133kl+fj5jx44N+Hqwy32XlJQQdogVGYYOHVo9wR6GnUEcpc17dvPqyicIK2nG2AvucTocY0wNdNZZZxEXF1eptv369eO+++7j9NNP58UXX+Tjjz+mT58+9OrVi3PPPZcdO3YAMG7cOP76V996b1dccQW33norp5xyCu3atStbqqOq7AziKF039UG8rlweOulJYiOjnA7HGFPBE/Of4KfdPx2xXmmd0rGIw+mU2Im70u+qcmzBys3NZfbs2QBkZ2dz4YUXIiK88sorPPPMMzzxxBMHtdmxYwfff/89y5Yt49JLL62WMwxLEEfh5XnTyCz5hq4NhjC0yylOh2OMqaOGDx9e9vzXX3/l0ksvZdu2bezfv5+OHTsGbHPRRRchInTv3p3NmzdXSxwhTRAiMhB4HnAD41T1oHV4ReRSYBSg+O49fbm/vDUwDmjlf22Qqv4SyngPZ9veHF5e/jhumjL2gnudCsMYcwTBftOvybOYGjRoUPb8pptu4t5772XQoEF8+eWXPP544OXMIyN/myxTXWvshSxBiIgbeAk4B8gEMkTkE1VdWa5OB+Ae4FRVzRaRJuV28RbwqKrOFJFYwBuqWINx3aej8LpzuL/3SzSKinEyFGNMPbJnzx5atmyJqvLmm28e0/cO5SB1OrBWVderahEwCRhSoc71wEuqmg2gqjsAROQEIExVZ/rL81Q1P4SxHta4BTPYWDyLzjGDubTbaU6FYYypJU477TSGDRvGrFmzSElJYfr06ZXe16hRoxg6dChnnHEGTZs2rcYojyyUXUwtgU3ltjOBinNCOwKIyPf4uqFGqeoX/vIcEfkvkAp8Cdytqp4QxhvQjrxcXvjxUdwk89oF9x3rtzfG1BJ5eXllz+fMmRNUm7Zt27J8+fIDyr777rsDti+55JIDbkFa6rrrrit7PnHixEPGUhWhTBASoKxix1gY0AE4E0gB5ohIV3/5aUAv4FfgPeBqYPwBbyAyEhgJ0Lp16+qLvJzrPn0Irzube3q9QHx07JEbGGNqhZo49lDThLKLKRPfAHOpFGBLgDofq2qxqm4AVuNLGJnAYn/3VAnwEXBixTdQ1VdVNU1V05KTk6v9ACYs+pINRTPoGD2QET3OrPb9G2NMTRbKBJEBdBCRVBGJAIYDn1So8xHQH0BEGuPrWlrvb5sgIqWf+gOAlRxDu/P38uziR3F5khh3wQPH8q2NMaZGCFmC8H/zvxmYDqwCJqvqChEZLSIX+qtNB7JEZCXwNXCnqmb5xxruAGaJyDJ83VWvhSrWQK795GG8Ybu4vef9JMZY15Ixpv4J6XUQqjoNmFah7IFyzxW43f+o2HYm0D2U8R3KO0u+ZU3hF7SPOpc/nni2EyEYY4zj7ErqCrLz9/HkwtG4JYHxFzzodDjGmBDZeOVVALR5+y2HI6m5bLG+Cq7/9FG8YTv4S497adygodPhGGNqidLlvpcsWcLJJ59Mly5d6N69O++9995BdW2571rovaVz+KlgKm0jzuLa3uc5HY4xphaKiYnhrbfeokOHDmzZsoXevXtz3nnnER8fX1antiz3bQnCb09hPmMyHsIljRh/4SinwzHG1FLlF9Nr0aIFTZo0YefOnQckiMPp168fZ5xxBnPmzOHiiy8mNTWVxx57jKKiIpKTk5k4cSJNmjRh3LhxLF++nOeee44rrriCpKQkMjIy2LZtG88884yt5lpdLhv7A2uK38MTt52bT3iCprHB/UEaY2qebY89xv5VR17uu/AnX53SsYjDiezciWb3Hv0infPnz6eoqIj27dsfVTtb7rsGySpZS1HsV7QOO5MbThrkdDjGmDpg69atXHnllbz55pu4XEc33FsvlvuuDfbuL2BrxOtQEsu4ix9yOhxjTBUF+00/lLOYcnNzGTx4MI888gh9+/Y96vY1Zbnvej+L6aedmwElrDiVFg0TnQ7HGFPLFRUVMXToUK666iqGDRtW5f05udx3vT+DOCnlOE7wPIzLVe9/FcaYajB58mRmz55NVlYWEyZMAGDChAn07NmzUvsrXe47JSWF9PR0tm7dWo3RHp5U16mI09LS0nTBggWVanvZ2B8AeO+Gk6szJGPMMbJq1So6d+58VG3q44VygX5PIrJQVdMC1bevzVhiMKY+qk+JobLq/RiEMcaYwCxBGGOMCcgShDGmTqgr46mhUpnfjyUIY0ytFxUVRVZWliWJQ1BVsrKyiIqKOqp2NkhtjKn1UlJSyMzMZOfOnU6HUmNFRUWRkpJyVG0sQRhjar3w8HBSU1OdDqPOsS4mY4wxAVmCMMYYE5AlCGOMMQHVmaU2RGQnsLEKu2gM7KqmcJxUV44D7FhqqrpyLHXlOKBqx9JGVZMDvVBnEkRViciCQ61HUpvUleMAO5aaqq4cS105DgjdsVgXkzHGmIAsQRhjjAnIEsRvXnU6gGpSV44D7FhqqrpyLHXlOCBEx2JjEMYYYwKyMwhjjDEBWYLwE5GHRWSpiCwRkRki0sLpmCpLRJ4SkZ/8x/OhiMQ7HVNlicgwEVkhIl4RqXUzTkRkoIisFpG1InK30/FUhYi8LiI7RGS507FUhYi0EpGvRWSV/+/WrU7HVFkiEiUi80XkR/+xPFSt+7cuJh8Raaiquf7nfwFOUNUbHQ6rUkTkXOArVS0RkScAVPUuh8OqFBHpDHiBscAdqlq5+8o6QETcwM/AOUAmkAGMUNWVjgZWSSJyOpAHvKWqXZ2Op7JEpDnQXFUXiUgcsBC4qDb+uYiIAA1UNU9EwoHvgFtVdW517N/OIPxKk4NfA6DWZk5VnaGqJf7NucDRLeFYg6jqKlVd7XQclZQOrFXV9apaBEwChjgcU6Wp6mxgt9NxVJWqblXVRf7ne4FVQEtno6oc9cnzb4b7H9X22WUJohwReVRENgF/AB5wOp5q8ifgc6eDqKdaApvKbWdSSz+I6ioRaQv0AuY5G0nliYhbRJYAO4CZqlptx1KvEoSIfCkiywM8hgCo6n2q2gp4B7jZ2WgP70jH4q9zH1CC73hqrGCOpZaSAGW19sy0rhGRWOAD4K8VehBqFVX1qGpPfD0F6SJSbd1/9ep+EKp6dpBV/wN8BjwYwnCq5EjHIiJ/BM4HztIaPtB0FH8utU0m0KrcdgqwxaFYTDn+/voPgHdU9b9Ox1MdVDVHRL4BBgLVMpGgXp1BHI6IdCi3eSHwk1OxVJWIDATuAi5U1Xyn46nHMoAOIpIqIhHAcOATh2Oq9/wDu+OBVar6T6fjqQoRSS6dpSgi0cDZVONnl81i8hORD4Dj8c2Y2QjcqKqbnY2qckRkLRAJZPmL5tbiGVlDgX8ByUAOsERVz3M2quCJyCDgOcANvK6qjzocUqWJyLvAmfhWDt0OPKiq4x0NqhJEpB8wB1iG7987wL2qOs25qCpHRLoDb+L7++UCJqvq6GrbvyUIY4wxgVgXkzHGmIAsQRhjjAnIEoQxxpiALEEYY4wJyBKEMcaYgCxBGHMURCTvyLUO236KiLTzP48VkbEiss6/EudsEekjIhH+5/XqQlZT81iCMOYYEZEugFtV1/uLxuFb/K6DqnYBrgYa+xf2mwVc5kigxvhZgjCmEsTnKf+aUctE5DJ/uUtE/u0/I5gqItNE5Pf+Zn8APvbXaw/0Ae5XVS+Af9XXz/x1P/LXN8YxdgprTOVcDPQEeuC7sjhDRGYDpwJtgW5AE3xLSb/ub3Mq8K7/eRd8V4V7DrH/5cBJIYncmCDZGYQxldMPeNe/kuZ24Ft8H+j9gPdV1auq24Cvy7VpDuwMZuf+xFHkv6GNMY6wBGFM5QRayvtw5QAFQJT/+Qqgh4gc7t9gJFBYidiMqRaWIIypnNnAZf6btSQDpwPz8d3y8RL/WERTfIvblVoFHAegquuABcBD/tVFEZEOpffAEJEkYKeqFh+rAzKmIksQxlTOh8BS4EfgK+Dv/i6lD/DdB2I5vvtozwP2+Nt8xoEJ4zqgGbBWRJYBr/Hb/SL6A7VudVFTt9hqrsZUMxGJ9d9EPgnfWcWpqrrNv17/1/7tQw1Ol+7jv8A9tfh+3KYOsFlMxlS/qf6buEQAD/vPLFDVAhF5EN99qX89VGP/zYU+suRgnGZnEMYYYwKyMQhjjDEBWYIwxhgTkCUIY4wxAVmCMMYYE5AlCGOMMQFZgjDGGBPQ/wO25/Or15HXswAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘制交叉验证误差曲线 - 正确率\n",
    "test_means = grid_accuracy.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_accuracy.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_accuracy.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_accuracy.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "n_Cs = len(Cs)\n",
    "number_penalties = len(penalties)\n",
    "test_scores = np.array(test_means).reshape(n_Cs, number_penalties)\n",
    "train_scores = np.array(train_means).reshape(n_Cs, number_penalties)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs, number_penalties)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs, number_penalties)\n",
    "\n",
    "x_axis = np.log10(Cs)  \n",
    "for i, value in enumerate(penalties):\n",
    "    plt.errorbar(x_axis, test_scores[:, i], yerr=test_stds[:, i], label=penalties[i] + ' Test')\n",
    "    print(penalties[i].capitalize() + ' Test')\n",
    "    for j in range(0, n_Cs):\n",
    "        print('C={} \\t - Accuracy={} '.format(Cs[j], test_scores[j,i]))\n",
    "\n",
    "for i, value in enumerate(penalties):\n",
    "    plt.errorbar(x_axis, train_scores[:, i], yerr=train_stds[:, i], label=penalties[i] + ' Train')\n",
    "    print('\\n' + penalties[i].capitalize() + ' Train')\n",
    "    for j in range(0, n_Cs):\n",
    "        print('C={} \\t - Accuracy={} '.format(Cs[j], train_scores[j,i]))\n",
    "    \n",
    "plt.legend()\n",
    "plt.xlabel(\"log(C)\")                                                                                                      \n",
    "plt.ylabel(\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由上面数据列表与曲线图可知：给定了 L1 正则和 L2 正则下、不同正则参数 C 对应的模型在训练集与测试集上的正确率。  \n",
    "可以看出在训练集上 C 越大（正则越小）的模型性能越好，但在测试集上当 C=0.1 时性能最好（L2正则）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle as cPickle\n",
    "\n",
    "cPickle.dump(grid_neg_log.best_estimator_, open(\"neglogloss_l1.pkl\", 'wb'))\n",
    "cPickle.dump(grid_accuracy.best_estimator_, open(\"accuracy_l2.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
